Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Weird Replication exception


+
Asaf Mesika 2013-06-02, 05:20
+
shashwat shriparv 2013-06-02, 14:21
+
Ted Yu 2013-06-02, 14:46
Copy link to this message
-
Re: Weird Replication exception
I use 0.94.7.
Is 0.94.8 production ready?

So in summary I have two issues:
1. Clocks are out of sync
2. I need to upgrade to 0.94.8 to avoid seeing this WARN messages?

On Jun 2, 2013, at 5:46 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> What is the HBase version you're using ?
>
> In another thread, I mentioned this:
>
> There was a recently integrated JIRA (0.94.8):
> HBASE-7122 Proper warning message when opening a log file with no entries
> (idle cluster)
>
> Does the HBase you're using contain HBASE-7122 ?
>
> Cheers
>
> On Sat, Jun 1, 2013 at 10:20 PM, Asaf Mesika <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I have a weird error in a cluster I'm checking Replication with.
>>
>> I have two clusters set up, each on its own DC (different continents). Each
>> has 1 master, and 3 RS.
>>
>> I've done all required setup, started replication and pushed in some data
>> into the master. I had an issue where the slave (peer) cluster went dead
>> (all RS failed contacting the master), thus replication couldn't work. This
>> happened right before the weekend, so it was out for 3 days.
>>
>> Now I'm back in the office - got slave cluster back up (just the RS), and I
>> got some nasty exception in one of the RS of the master cluster:
>>
>> 2013-06-02 04:40:45,903 INFO
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting
>> 0 rs from peer cluster # c
>> 2013-06-02 04:40:45,903 INFO
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Slave
>> cluster looks down: c has 0 region servers
>> 2013-06-02 04:40:46,903 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Since
>> we are unable to replicate, sleeping 1000 times 10
>> 2013-06-02 04:40:57,019 INFO
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting
>> 0 rs from peer cluster # c
>> 2013-06-02 04:40:57,019 INFO
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Slave
>> cluster looks down: c has 0 region servers
>> 2013-06-02 04:40:58,019 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Since
>> we are unable to replicate, sleeping 1000 times 10
>> 2013-06-02 04:41:08,134 INFO
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting
>> 1 rs from peer cluster # c
>> 2013-06-02 04:41:08,134 INFO
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Choosing peer a72-246-95-86,60020,1370147274693
>> 2013-06-02 04:41:08,672 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Replicating 1
>> 2013-06-02 04:41:08,971 INFO
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager:
>> Going to report log #172.25.98.74%2C60020%2C1369903540894.1369921571842 for
>> position
>> 49882 in hdfs://
>>
>> 172.25.98.73:8020/hbase/.logs/172.25.98.74,60020,1369903540894/172.25.98.74%2C60020%2C1369903540894.1369921571842
>> 2013-06-02 04:41:08,973 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Replicated in total: 1
>> 2013-06-02 04:41:08,973 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
>> log for replication 172.25.98.74%2C60020%2C1369903540894.1369921571842 at
>> 49882
>> 2013-06-02 04:41:08,977 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> currentNbOperations:0 and seenEntries:4 and size: 0
>> 2013-06-02 04:41:08,977 INFO
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager:
>> Going to report log #172.25.98.74%2C60020%2C1369903540894.1369921571842 for
>> position
>> 69368 in hdfs://
>>
>> 172.25.98.73:8020/hbase/.logs/172.25.98.74,60020,1369903540894/172.25.98.74%2C60020%2C1369903540894.1369921571842
>> 2013-06-02 04:41:08,978 INFO
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager:
>> Removing 0 logs in the list: []
>> 2013-06-02 04:41:08,978 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Nothing
+
Ted Yu 2013-06-02, 19:38
+
Himanshu Vashishtha 2013-06-03, 02:36
+
Asaf Mesika 2013-06-03, 05:18