Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Weird Replication exception


Copy link to this message
-
Re: Weird Replication exception
Ted Yu 2013-06-02, 19:38
bq. Is 0.94.8 production ready?

I think so. Lars released 0.94.8 Friday evening.

On Sun, Jun 2, 2013 at 12:26 PM, Asaf Mesika <[EMAIL PROTECTED]> wrote:

> I use 0.94.7.
> Is 0.94.8 production ready?
>
> So in summary I have two issues:
> 1. Clocks are out of sync
> 2. I need to upgrade to 0.94.8 to avoid seeing this WARN messages?
>
> On Jun 2, 2013, at 5:46 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > What is the HBase version you're using ?
> >
> > In another thread, I mentioned this:
> >
> > There was a recently integrated JIRA (0.94.8):
> > HBASE-7122 Proper warning message when opening a log file with no entries
> > (idle cluster)
> >
> > Does the HBase you're using contain HBASE-7122 ?
> >
> > Cheers
> >
> > On Sat, Jun 1, 2013 at 10:20 PM, Asaf Mesika <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi,
> >>
> >> I have a weird error in a cluster I'm checking Replication with.
> >>
> >> I have two clusters set up, each on its own DC (different continents).
> Each
> >> has 1 master, and 3 RS.
> >>
> >> I've done all required setup, started replication and pushed in some
> data
> >> into the master. I had an issue where the slave (peer) cluster went dead
> >> (all RS failed contacting the master), thus replication couldn't work.
> This
> >> happened right before the weekend, so it was out for 3 days.
> >>
> >> Now I'm back in the office - got slave cluster back up (just the RS),
> and I
> >> got some nasty exception in one of the RS of the master cluster:
> >>
> >> 2013-06-02 04:40:45,903 INFO
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Getting
> >> 0 rs from peer cluster # c
> >> 2013-06-02 04:40:45,903 INFO
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Slave
> >> cluster looks down: c has 0 region servers
> >> 2013-06-02 04:40:46,903 DEBUG
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Since
> >> we are unable to replicate, sleeping 1000 times 10
> >> 2013-06-02 04:40:57,019 INFO
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Getting
> >> 0 rs from peer cluster # c
> >> 2013-06-02 04:40:57,019 INFO
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Slave
> >> cluster looks down: c has 0 region servers
> >> 2013-06-02 04:40:58,019 DEBUG
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Since
> >> we are unable to replicate, sleeping 1000 times 10
> >> 2013-06-02 04:41:08,134 INFO
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Getting
> >> 1 rs from peer cluster # c
> >> 2013-06-02 04:41:08,134 INFO
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> >> Choosing peer a72-246-95-86,60020,1370147274693
> >> 2013-06-02 04:41:08,672 DEBUG
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> >> Replicating 1
> >> 2013-06-02 04:41:08,971 INFO
> >>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager:
> >> Going to report log #172.25.98.74%2C60020%2C1369903540894.1369921571842
> for
> >> position
> >> 49882 in hdfs://
> >>
> >>
> 172.25.98.73:8020/hbase/.logs/172.25.98.74,60020,1369903540894/172.25.98.74%2C60020%2C1369903540894.1369921571842
> >> 2013-06-02 04:41:08,973 DEBUG
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> >> Replicated in total: 1
> >> 2013-06-02 04:41:08,973 DEBUG
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Opening
> >> log for replication 172.25.98.74%2C60020%2C1369903540894.1369921571842
> at
> >> 49882
> >> 2013-06-02 04:41:08,977 DEBUG
> >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> >> currentNbOperations:0 and seenEntries:4 and size: 0
> >> 2013-06-02 04:41:08,977 INFO
> >>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager:
> >> Going to report log #172.25.98.74%2C60020%2C1369903540894.1369921571842
> for
> >> position
> >> 69368 in hdfs://