Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Weird Replication exception


+
Asaf Mesika 2013-06-02, 05:20
+
shashwat shriparv 2013-06-02, 14:21
+
Ted Yu 2013-06-02, 14:46
+
Asaf Mesika 2013-06-02, 19:26
+
Ted Yu 2013-06-02, 19:38
Copy link to this message
-
Re: Weird Replication exception
Hey Asaf,

It looks like you only need 7122. Either upgrade, or you could also patch
it up.

Syncing up the master and slave cluster is also advised, but that stands
good in case you are using master-master replication.
bq. 172.25.98.74,60020,
1369903540894/172.25.98.74
%2C60020%2C1369903540894.1369925171871

bq.    Meaning I lost data

Did the log whose znode you deleted had any data?
You can do a cat to see if there is any data on it. You could copy-table
for that time range, (or a hacky way is to re-create a znode for that log
under a regionserver noticing the format of its current log znodes, and let
the replicationSource pick it up in its normal run).

Thanks,
Himanshu

On Sun, Jun 2, 2013 at 12:38 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> bq. Is 0.94.8 production ready?
>
> I think so. Lars released 0.94.8 Friday evening.
>
> On Sun, Jun 2, 2013 at 12:26 PM, Asaf Mesika <[EMAIL PROTECTED]>
> wrote:
>
> > I use 0.94.7.
> > Is 0.94.8 production ready?
> >
> > So in summary I have two issues:
> > 1. Clocks are out of sync
> > 2. I need to upgrade to 0.94.8 to avoid seeing this WARN messages?
> >
> > On Jun 2, 2013, at 5:46 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > What is the HBase version you're using ?
> > >
> > > In another thread, I mentioned this:
> > >
> > > There was a recently integrated JIRA (0.94.8):
> > > HBASE-7122 Proper warning message when opening a log file with no
> entries
> > > (idle cluster)
> > >
> > > Does the HBase you're using contain HBASE-7122 ?
> > >
> > > Cheers
> > >
> > > On Sat, Jun 1, 2013 at 10:20 PM, Asaf Mesika <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> I have a weird error in a cluster I'm checking Replication with.
> > >>
> > >> I have two clusters set up, each on its own DC (different continents).
> > Each
> > >> has 1 master, and 3 RS.
> > >>
> > >> I've done all required setup, started replication and pushed in some
> > data
> > >> into the master. I had an issue where the slave (peer) cluster went
> dead
> > >> (all RS failed contacting the master), thus replication couldn't work.
> > This
> > >> happened right before the weekend, so it was out for 3 days.
> > >>
> > >> Now I'm back in the office - got slave cluster back up (just the RS),
> > and I
> > >> got some nasty exception in one of the RS of the master cluster:
> > >>
> > >> 2013-06-02 04:40:45,903 INFO
> > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > Getting
> > >> 0 rs from peer cluster # c
> > >> 2013-06-02 04:40:45,903 INFO
> > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > Slave
> > >> cluster looks down: c has 0 region servers
> > >> 2013-06-02 04:40:46,903 DEBUG
> > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > Since
> > >> we are unable to replicate, sleeping 1000 times 10
> > >> 2013-06-02 04:40:57,019 INFO
> > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > Getting
> > >> 0 rs from peer cluster # c
> > >> 2013-06-02 04:40:57,019 INFO
> > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > Slave
> > >> cluster looks down: c has 0 region servers
> > >> 2013-06-02 04:40:58,019 DEBUG
> > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > Since
> > >> we are unable to replicate, sleeping 1000 times 10
> > >> 2013-06-02 04:41:08,134 INFO
> > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > Getting
> > >> 1 rs from peer cluster # c
> > >> 2013-06-02 04:41:08,134 INFO
> > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > >> Choosing peer a72-246-95-86,60020,1370147274693
> > >> 2013-06-02 04:41:08,672 DEBUG
> > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > >> Replicating 1
> > >> 2013-06-02 04:41:08,971 INFO
> > >>
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager:
> > >> Going to report log
> #172.25.98.74%2C60020%2C1369903540894.1369921571842
+
Asaf Mesika 2013-06-03, 05:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB