Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Never ending distributed log split


Copy link to this message
-
Re: Never ending distributed log split
Jean-Marc Spaggiari 2012-08-03, 13:43
Here us the complete log. And seems it's every 30 seconds and not
every 20 seconds...

http://pastebin.com/gMiURnnj

2012/8/3, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> Hi,
>
> I'm using HBase 0.94.0.
>
> I stopped the cluster for some maintenance, and I'm have some troubles
> to restart it.
>
> I'm getting one line every about
>
> Start Time Description State Status
> Fri Aug 03 08:59:54 EDT 2012 Doing distributed log split in
> [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting,
> hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting,
> hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting,
> hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting]
> RUNNING (since 3sec ago) Waiting for distributed tasks to finish.
> scheduled=1 done=0 error=0 (since 0sec ago)
>
> If I let it run, it will run like that for hours. Adding lines and
> lines and lines until I stop it.
>
>
> On the master logs, I can see that:
> 2012-08-03 09:02:49,788 INFO
> org.apache.hadoop.hbase.master.SplitLogManager: task
> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297
> entered state err node4,60020,1343998592129
> 2012-08-03 09:02:49,788 WARN
> org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297
> 2012-08-03 09:02:49,788 WARN
> org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
> logs in
> [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting,
> hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting,
> hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting,
> hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting]
> installed = 1 but only 0 done