Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Never ending "Doing distributed log split" task.,


Copy link to this message
-
Never ending "Doing distributed log split" task.,
I have restart my cluster and I'm now waiting for this task to end:

Doing distributed log split in
[hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting]

It's running fir now 30 minutes. There was nothing running on the cluster.
No reads, no writes, nothing, for days...

I got that on the logs:

2013-08-29 11:36:10,862 WARN
org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting/node1%2C60020%2C1377789460683.1377789462024
interrupted, resigning
java.io.InterruptedIOException
    at
org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:136)
    at
org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:54)
    at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:780)
    at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:414)
    at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:381)
    at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:112)
    at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:280)
    at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
    at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.InterruptedException: sleep interrupted
    at java.lang.Thread.sleep(Native Method)
    at
org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:118)
    ... 9 more
2013-08-29 11:36:10,950 WARN
org.apache.hadoop.hbase.regionserver.SplitLogWorker: Interrupted while
trying to assert ownership of
/hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1377789460683-splitting%2Fnode1%252C60020%252C1377789460683.1377789462024
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1160)
    at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:361)
    at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.attemptToOwnTask(SplitLogWorker.java:346)
    at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
    at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
    at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
    at java.lang.Thread.run(Thread.java:722)
I'm not 100% what is causing that. I have restarted it and still getting
the same result.

Any hint?

Thanks,

JM
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB