Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> [Question: Replication] how to detect whether there is a replication pending?


+
Demai Ni 2013-09-18, 16:33
+
Jean-Daniel Cryans 2013-09-18, 17:38
+
Demai Ni 2013-09-18, 17:48
+
Demai Ni 2013-09-19, 00:52
Copy link to this message
-
Re: [Question: Replication] how to detect whether there is a replication pending?
hi, J-D and all,

still the same question. I thought I fix it before, while I was wrong.

in ReplicationSource, I used this code:
    float opsRate = this.metrics.shippedOpsRate.getPreviousIntervalValue();
    float batchRate this.metrics.shippedBatchesRate.getPreviousIntervalValue();
    float logFilteredRate this.metrics.logEditsFilteredRate.getPreviousIntervalValue();
    float logEditsReadRate this.metrics.logEditsReadRate.getPreviousIntervalValue();

but above values are already 0.0, even the logs shows some Edits are being
pushed over.

a bit background about what I am doing as the original question was asked a
month ago:

"(hbase 0.94.9) I am looking for a way to flag that NO replication is
pending. For example of a simple Master-Slave setup for table t1. At a
given point of time, can I tell whether there are still Edits pending for
replication?  checking the sizeOfLogQueue won't work ash
metrics.sizeOfLogQueue=0, there may still has one Log being processed for
replication with worse case of 64MB pending.  Hopefully the method will
works even when sink is offline"

many  thanks for your suggestions

Demai
On Wed, Sep 18, 2013 at 10:38 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> Yeah, with HLog queue you can be a whole 64MB in the worst case :)
>
> Pretty hard to say for a single table what the status is, but if it was
> cluster-wide you'd just have to look at the shipped ops/batch rates (if it
> sits at 0, you're not replicating, unless there's some problem preventing
> replication).
>
> J-D
>
>
> On Wed, Sep 18, 2013 at 9:33 AM, Demai Ni <[EMAIL PROTECTED]> wrote:
>
> > hi, folks,
> >
> > I am looking for a way to flag that NO replication is pending. For
> example
> > of a simple Master-Slave setup for table t1. At a given point of time,
> can
> > I tell whether there are still Edits pending for replication? I am
> thinking
> > about using replicationsource.metrics.sizeOfLogQueue. The only concern is
> > that even with metrics.sizeOfLogQueue=0, there may still has one Log
> being
> > processed for replication. Is my understanding correct?
> >
> > Thanks
> >
> > Demai
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB