Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Never ending distributed log split


Copy link to this message
-
Never ending distributed log split
Jean-Marc Spaggiari 2012-08-03, 13:33
Hi,

I'm using HBase 0.94.0.

I stopped the cluster for some maintenance, and I'm have some troubles
to restart it.

I'm getting one line every about

Start Time Description State Status
Fri Aug 03 08:59:54 EDT 2012 Doing distributed log split in
[hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting,
hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting,
hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting,
hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting,
hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting,
hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting,
hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting,
hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting,
hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting,
hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting,
hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting,
hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting,
hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting,
hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting,
hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting,
hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting,
hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting,
hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting,
hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting]
RUNNING (since 3sec ago) Waiting for distributed tasks to finish.
scheduled=1 done=0 error=0 (since 0sec ago)

If I let it run, it will run like that for hours. Adding lines and
lines and lines until I stop it.
On the master logs, I can see that:
2012-08-03 09:02:49,788 INFO
org.apache.hadoop.hbase.master.SplitLogManager: task
/hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297
entered state err node4,60020,1343998592129
2012-08-03 09:02:49,788 WARN
org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
/hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297
2012-08-03 09:02:49,788 WARN
org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
logs in [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting,
hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting,
hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting,
hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting,
hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting,
hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting,
hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting,
hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting,
hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting,
hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting,
hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting,
hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting,
hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting,
hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting,
hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting,
hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting,
hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting,
hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting,
hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting]
installed = 1 but only 0 done
2012-08-03 09:02:49,788 WARN
org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting of
[latitude,60020,1343908057839, latitude,60020,1343998595290,
node1,60020,1343908057567, node1,60020,1343939284240,
node1,60020,1343998593757, node2,60020,1343908059614,
node2,60020,1343939286369, node2,60020,1343998595830,
node3,60020,1343908054414, node3,60020,1343939282294,
node3,60020,1343998590612, node4,60020,1343908056186,
node4,60020,1343939282889, node4,60020,1343998592129,
node5,60020,1343908059158, node5,60020,1343998594856,
phenom,60020,1343908053256, phenom,60020,1343939281065,
phenom,60020,1343998580375]
java.io.IOException: error or interrupt while splitting logs in
[hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting,
hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting,
hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting,
hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting,
hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting,
hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting,
hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting,
hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting,
hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting,
hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting,
hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting,
hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting,
hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting,
hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting,
hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting,
hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting,
hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting,
hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting,
hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting]
Task = installed = 1 done = 0 error = 1
        at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:269)
        at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:277)
        at org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAf