Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Never ending distributed log split


Copy link to this message
-
Re: Never ending distributed log split
Here us the complete log. And seems it's every 30 seconds and not
every 20 seconds...

http://pastebin.com/gMiURnnj

2012/8/3, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> Hi,
>
> I'm using HBase 0.94.0.
>
> I stopped the cluster for some maintenance, and I'm have some troubles
> to restart it.
>
> I'm getting one line every about
>
> Start Time Description State Status
> Fri Aug 03 08:59:54 EDT 2012 Doing distributed log split in
> [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting,
> hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting,
> hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting,
> hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting]
> RUNNING (since 3sec ago) Waiting for distributed tasks to finish.
> scheduled=1 done=0 error=0 (since 0sec ago)
>
> If I let it run, it will run like that for hours. Adding lines and
> lines and lines until I stop it.
>
>
> On the master logs, I can see that:
> 2012-08-03 09:02:49,788 INFO
> org.apache.hadoop.hbase.master.SplitLogManager: task
> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297
> entered state err node4,60020,1343998592129
> 2012-08-03 09:02:49,788 WARN
> org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297
> 2012-08-03 09:02:49,788 WARN
> org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
> logs in
> [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting,
> hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting,
> hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting,
> hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting,
> hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting,
> hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting,
> hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting,
> hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting,
> hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting]
> installed = 1 but only 0 done
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB