Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Never ending distributed log split


+
Jean-Marc Spaggiari 2013-06-02, 15:09
+
Stack 2013-06-03, 04:35
+
Ted Yu 2013-06-02, 15:46
Copy link to this message
-
Re: Never ending distributed log split
Jean-Marc Spaggiari 2013-06-02, 17:05
I'm using 0.94.7 since I did not get the chance to deploye the last RC...

I will wait for some more feedback regarding the option (delete or
rename) and most probably will open a JIRA.

Regardeing recovered.editsI don't have this file anymore, but I just
found another one which is blocking some other splits:
hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls
/hbase/work_proposed/4a45c6fed3578cd6f547c6fc58bad221/
Found 5 items
drwxr-xr-x   - hbase supergroup          0 2013-06-01 16:56
/hbase/work_proposed/4a45c6fed3578cd6f547c6fc58bad221/.oldlogs
-rw-r--r--   3 hbase supergroup        855 2013-06-01 16:56
/hbase/work_proposed/4a45c6fed3578cd6f547c6fc58bad221/.regioninfo
drwxr-xr-x   - hbase supergroup          0 2013-06-01 17:48
/hbase/work_proposed/4a45c6fed3578cd6f547c6fc58bad221/@
drwxr-xr-x   - hbase supergroup          0 2013-06-01 16:56
/hbase/work_proposed/4a45c6fed3578cd6f547c6fc58bad221/a
-rw-r--r--   3 hbase supergroup       5375 2013-06-01 15:43
/hbase/work_proposed/4a45c6fed3578cd6f547c6fc58bad221/recovered.edits

And the date/time seems to match when I faced the 2 power outages yesterday...

JM

2013/6/2 Ted Yu <[EMAIL PROTECTED]>:
> Can you search for 1d44b0630ed7785106a87a2bd4993551/recovered.edits to see
> when it was created ?
> Namenode log would be a good place to start with.
>
> bq. we can also rename it so if really required we can replay it later?
>
> The above is a better way of handling the situation.
>
> What version of HBase are you using ?
>
> Cheers
>
> On Sun, Jun 2, 2013 at 8:09 AM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
>> wrote:
>
>> My HBase was in a bad state recently. HBCK did a slow but good job and
>> everything is now almost stable. However, I still have one log split
>> which is not working. Every minute, the SplitLogManager try to split
>> the log, fails, and retry. It's always the same file. It's assigned to
>> different nodes, but all failed, and it's starting again and again.
>>
>>
>> 2013-06-02 10:44:20,298 DEBUG
>> org.apache.hadoop.hbase.master.SplitLogManager: Scheduling batch of
>> logs to split
>> 2013-06-02 10:44:20,298 INFO
>> org.apache.hadoop.hbase.master.SplitLogManager: started splitting logs
>> in [hdfs://node3:9000/hbase/.logs/node7,60020,1370118961527-splitting]
>> 2013-06-02 10:44:20,298 DEBUG
>> org.apache.hadoop.hbase.master.SplitLogManager: wait for status of
>> task
>> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode7%2C60020%2C1370118961527-splitting%2Fnode7%252C60020%252C1370118961527.1370122562614
>> to change to DELETED
>> 2013-06-02 10:44:20,315 DEBUG
>> org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback:
>> deleted
>> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode7%2C60020%2C1370118961527-splitting%2Fnode7%252C60020%252C1370118961527.1370122562614
>> 2013-06-02 10:44:20,329 DEBUG
>> org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task
>> at znode
>> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode7%2C60020%2C1370118961527-splitting%2Fnode7%252C60020%252C1370118961527.1370122562614
>> 2013-06-02 10:44:20,341 DEBUG
>> org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task
>> at znode
>> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode7%2C60020%2C1370118961527-splitting%2Fnode7%252C60020%252C1370118961527.1370129764666
>> 2013-06-02 10:44:20,344 DEBUG
>> org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired
>>
>> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode7%2C60020%2C1370118961527-splitting%2Fnode7%252C60020%252C1370118961527.1370122562614
>> ver = 0
>> 2013-06-02 10:44:20,346 DEBUG
>> org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired
>>
>> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode7%2C60020%2C1370118961527-splitting%2Fnode7%252C60020%252C1370118961527.1370129764666
>> ver = 0
>> 2013-06-02 10:44:20,384 INFO
>> org.apache.hadoop.hbase.master.SplitLogManager: task
>>
>