Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Cyclic Replication Issue: some data are missing in the replication for intensive write


Copy link to this message
-
Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Hi Lars:

I'm using hbase 0.92.1 and Hadoop 1.0.1.
Yes, you are right. I'm replicating from cluster A to cluster B only with cyclic replication configured. Eventually I will test replicating cluster A to cluster B and vice versa with high intensive write workload but if this replication doesn't work for one way, we need to think about other solutions.

No data loss in cluster A for sure.

Best Regards,

Jerry

Sent from my iPad

On 2012-04-20, at 15:34, lars hofhansl <[EMAIL PROTECTED]> wrote:

> Hi Jerry,
>
> which version of HBase are you using?
>
> You are not using cyclic backup, that needs >2 clusters. I assume you're just replicating from one cluster to another, right?
>
> There is never data loss in Cluster A?
>
> -- Lars
>
>
> ----- Original Message -----
> From: Jerry Lam <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Friday, April 20, 2012 5:38 AM
> Subject: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
>
> Hi HBase community:
>
> We have been testing cyclic replication for 1 week. The basic functionality seems to work as described in the document however when we started to increase the write workload, the replication starts to miss data (i.e. some data are not replicated to the other cluster). We have narrowed down to a scenario that we can reproduce the problem quite consistently and here it is:
>
> -----------------------------
> Setup:
> - We have setup 2 clusters (cluster A and cluster B)with identical size in terms of number of nodes and configuration, 3 regionservers sit on top of 3 datanodes.
> - Cyclic replication is enabled.
>
> - We use YCSB to generate load to hbase the workload is very similar to workloada:
>
> recordcount=200000
> operationcount=200000
> workload=com.yahoo.ycsb.workloads.CoreWorkload
> fieldcount=1
> fieldlength=25000
>
> readallfields=true
> writeallfields=true
>
> readproportion=0
> updateproportion=1
> scanproportion=0
> insertproportion=0
>
> requestdistribution=uniform
>
> - Records are inserted into Cluster A. After the benchmark is done and wait until all data are replicated to Cluster B, we used verifyrep mapreduce job for validation.
> - Data are deleted from both table (truncate 'tablename') before a new experiment is started.
>
> Scenario:
> when we increase the number of threads until it max out the throughput of the cluster, we saw some data are missing in Cluster B (total count != 200000) although cluster A clearly has them all. This happens even though we disabled region splitting in both clusters (it happens more often when region splits occur). To further having more control of what is happening, we then decided to disable the load balancer so the region (which is responsible for the replicating data) will not relocate to other regionserver during the benchmark. The situation improves a lot. We don't see any missing data in 5 continuous runs. Finally, we decided to move the region around from a regionserver to another regionserver during the benchmark to see if the problem will reappear and it did.
>
> We believe that the issue could be related to region splitting and load balancing during intensive write, the hbase replication strategy hasn't yet cover those corner cases.
>
> Can someone take a look of it and suggest some ways to workaround this?
>
> Thanks~
>
> Jerry
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB