Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Never ending distributed log split


Copy link to this message
-
Re: Never ending distributed log split
Me again ;)

I did some more investigation.

And I found that:

http://pastebin.com/Bedm6Ldy

Seems that no region is serving my logs. That's strange because all my
servers are up and fsck is telling me that FS is clean.

Can I just delete those files? What's the impact of such delete? I
don't really worrie about loosing some data. It's a test environment.
But I really need it to start again.

Thanks,

JM

2012/8/3, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> Here us the complete log. And seems it's every 30 seconds and not
> every 20 seconds...
>
> http://pastebin.com/gMiURnnj
>
> 2012/8/3, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
>> Hi,
>>
>> I'm using HBase 0.94.0.
>>
>> I stopped the cluster for some maintenance, and I'm have some troubles
>> to restart it.
>>
>> I'm getting one line every about
>>
>> Start Time Description State Status
>> Fri Aug 03 08:59:54 EDT 2012 Doing distributed log split in
>> [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting,
>> hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting,
>> hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting,
>> hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting,
>> hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting,
>> hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting,
>> hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting,
>> hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting,
>> hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting,
>> hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting,
>> hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting,
>> hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting,
>> hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting,
>> hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting,
>> hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting,
>> hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting,
>> hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting,
>> hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting,
>> hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting]
>> RUNNING (since 3sec ago) Waiting for distributed tasks to finish.
>> scheduled=1 done=0 error=0 (since 0sec ago)
>>
>> If I let it run, it will run like that for hours. Adding lines and
>> lines and lines until I stop it.
>>
>>
>> On the master logs, I can see that:
>> 2012-08-03 09:02:49,788 INFO
>> org.apache.hadoop.hbase.master.SplitLogManager: task
>> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297
>> entered state err node4,60020,1343998592129
>> 2012-08-03 09:02:49,788 WARN
>> org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
>> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297
>> 2012-08-03 09:02:49,788 WARN
>> org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
>> logs in
>> [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting,
>> hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting,
>> hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting,
>> hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting,
>> hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting,
>> hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting,
>> hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting,
>> hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting,
>> hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting,
>> hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting,
>> hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting,
>> hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting,
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB