Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Weird Replication exception


+
Asaf Mesika 2013-06-02, 05:20
+
shashwat shriparv 2013-06-02, 14:21
Copy link to this message
-
Re: Weird Replication exception
What is the HBase version you're using ?

In another thread, I mentioned this:

There was a recently integrated JIRA (0.94.8):
HBASE-7122 Proper warning message when opening a log file with no entries
(idle cluster)

Does the HBase you're using contain HBASE-7122 ?

Cheers

On Sat, Jun 1, 2013 at 10:20 PM, Asaf Mesika <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have a weird error in a cluster I'm checking Replication with.
>
> I have two clusters set up, each on its own DC (different continents). Each
> has 1 master, and 3 RS.
>
> I've done all required setup, started replication and pushed in some data
> into the master. I had an issue where the slave (peer) cluster went dead
> (all RS failed contacting the master), thus replication couldn't work. This
> happened right before the weekend, so it was out for 3 days.
>
> Now I'm back in the office - got slave cluster back up (just the RS), and I
> got some nasty exception in one of the RS of the master cluster:
>
> 2013-06-02 04:40:45,903 INFO
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting
> 0 rs from peer cluster # c
> 2013-06-02 04:40:45,903 INFO
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Slave
> cluster looks down: c has 0 region servers
> 2013-06-02 04:40:46,903 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Since
> we are unable to replicate, sleeping 1000 times 10
> 2013-06-02 04:40:57,019 INFO
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting
> 0 rs from peer cluster # c
> 2013-06-02 04:40:57,019 INFO
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Slave
> cluster looks down: c has 0 region servers
> 2013-06-02 04:40:58,019 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Since
> we are unable to replicate, sleeping 1000 times 10
> 2013-06-02 04:41:08,134 INFO
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting
> 1 rs from peer cluster # c
> 2013-06-02 04:41:08,134 INFO
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Choosing peer a72-246-95-86,60020,1370147274693
> 2013-06-02 04:41:08,672 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Replicating 1
> 2013-06-02 04:41:08,971 INFO
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager:
> Going to report log #172.25.98.74%2C60020%2C1369903540894.1369921571842 for
> position
>  49882 in hdfs://
>
> 172.25.98.73:8020/hbase/.logs/172.25.98.74,60020,1369903540894/172.25.98.74%2C60020%2C1369903540894.1369921571842
> 2013-06-02 04:41:08,973 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Replicated in total: 1
> 2013-06-02 04:41:08,973 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
> log for replication 172.25.98.74%2C60020%2C1369903540894.1369921571842 at
> 49882
> 2013-06-02 04:41:08,977 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> currentNbOperations:0 and seenEntries:4 and size: 0
> 2013-06-02 04:41:08,977 INFO
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager:
> Going to report log #172.25.98.74%2C60020%2C1369903540894.1369921571842 for
> position
>  69368 in hdfs://
>
> 172.25.98.73:8020/hbase/.logs/172.25.98.74,60020,1369903540894/172.25.98.74%2C60020%2C1369903540894.1369921571842
> 2013-06-02 04:41:08,978 INFO
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager:
> Removing 0 logs in the list: []
> 2013-06-02 04:41:08,978 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Nothing
> to replicate, sleeping 1000 times 1
> 2013-06-02 04:41:09,978 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
> log for replication 172.25.98.74%2C60020%2C1369903540894.1369921571842 at
> 69368
> 2013-06-02 04:41:09,980 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
+
Asaf Mesika 2013-06-02, 19:26
+
Ted Yu 2013-06-02, 19:38
+
Himanshu Vashishtha 2013-06-03, 02:36
+
Asaf Mesika 2013-06-03, 05:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB