Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> RS crash upon replication


Copy link to this message
-
Re: RS crash upon replication
Basically,

You had va-p-hbase-02 crash - that caused all the replication related data
in zookeeper to be moved to va-p-hbase-01 and have it take over for
replicating 02's logs. Now each region server also maintains an in-memory
state of whats in ZK, it seems like when you start up 01, its trying to
replicate the 02 logs underneath but its failing to because that data is
not in ZK. This is somewhat weird...

Can you open the zookeepeer shell and do

ls /hbase/replication/rs/va-p-hbase-01-c,60020,1369249873379

And give the output ?
On Wed, May 22, 2013 at 1:27 PM, [EMAIL PROTECTED] <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> This is bad ... and happened twice: I had my replication-slave cluster
> offlined. I performed quite a massive Merge operation on it and after a
> couple of hours it had finished and I returned it back online. At the same
> time, the replication-master RS machines crashed (see first crash
> http://pastebin.com/1msNZ2tH) with the first exception being:
>
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode > NoNode for
>
> /hbase/replication/rs/va-p-hbase-01-c,60020,1369233253404/1-va-p-hbase-01-c,60020,1369042378287-va-p-hbase-02-c,60020,1369042377731/va-p-hbase-01-c%2C60020%2C1369042378287.1369220050719
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>         at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
>         at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:354)
>         at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:846)
>         at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:898)
>         at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:892)
>         at
>
> org.apache.hadoop.hbase.replication.ReplicationZookeeper.writeReplicationStatus(ReplicationZookeeper.java:558)
>         at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:154)
>         at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:638)
>         at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:387)
>
> Before restarting the crashed RS's, I have applied a 'stop_replication'
> cmd. Then fired up the RS's again. They've started o.k. but once I've hit
> 'start_replication' they have crashed once again. The second crash log
> http://pastebin.com/8Nb5epJJ has the same initial exception
> (org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode). I've started the crash region servers again
> without replication and currently all is well, but I need to start
> replication asap.
>
> Does anyone have an idea what's going on and how can I solve it ?
>
> Thanks,
> Amit
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB