Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Recovering corrupt HLog files


Copy link to this message
-
Re: Recovering corrupt HLog files
This is interesting because I saw this happens in the past. Is walplayer can be back ported to 0.90.x?

Best Regards,

Jerry

Sent from my iPad

On 2012-06-30, at 16:34, Li Pi <[EMAIL PROTECTED]> wrote:

> Nope. It came out in 0.94 otoh.
>
> On Sat, Jun 30, 2012 at 12:29 PM, Bryan Beaudreault <
> [EMAIL PROTECTED]> wrote:
>
>> I should have mentioned in my initial email that I am operating on HBase
>> 0.90.4.  Is WALPlayer available in this version?  I am having trouble
>> finding it or anything similar.
>>
>> On Sat, Jun 30, 2012 at 1:14 PM, Li Pi <[EMAIL PROTECTED]> wrote:
>>
>>> WALPlayer will look at the timestamp. Replaying an older edit that has
>>> since been overwritten shouldn't change anything.
>>>
>>> On Sat, Jun 30, 2012 at 9:49 AM, Bryan Beaudreault <
>>> [EMAIL PROTECTED]
>>>> wrote:
>>>
>>>> They are all pretty large, around 40+mb.  Will the walplayer be smart
>>>> enough to only write edits that still look relevant (i.e. based on
>>>> timestamps of the edits vs timestamps of the versions in hbase)?
>> Writes
>>>> have been coming in since we recovered.
>>>>
>>>> On Sat, Jun 30, 2012 at 11:05 AM, Stack <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> On Sat, Jun 30, 2012 at 8:38 AM, Bryan Beaudreault
>>>>> <[EMAIL PROTECTED]> wrote:
>>>>>> 12/06/30 00:00:48 INFO wal.HLogSplitter: Got while parsing hlog
>>>>>>
>>>>>
>>>>
>>>
>> hdfs://my-namenode-ip-addr:8020/hbase/.logs/my-rs-ip-addr,60020,1338667719591/my-rs-ip-addr%3A60020.1340935453874.
>>>>>> Marking as corrupted
>>>>>>
>>>>>
>>>>> What size do these logs have?
>>>>>
>>>>>> We are back to stable operating now, and in trying to research
>> this I
>>>>> found
>>>>>> the hdfs://my-namenode-ip-addr:8020/hbase/.corrupt directory.
>> There
>>>> are
>>>>> 20
>>>>>> files listed there.
>>>>>>
>>>>>
>>>>> Ditto.
>>>>>
>>>>>> What are our options for tracking down and potentially recovering
>> any
>>>>> data
>>>>>> that was lost.  Or how can we even tell what was lost, if any?
>> Does
>>>> the
>>>>>> existence of these files pretty much guarantee data lost? There
>>> doesn't
>>>>>> seem to be much documentation on this.  From reading it seems like
>> it
>>>>> might
>>>>>> be possible that part of each of these files was recovered.
>>>>>>
>>>>>
>>>>> If size > 0, could try walplaying them:
>>>>> http://hbase.apache.org/book.html#walplayer
>>>>>
>>>>> St.Ack
>>>>>
>>>>
>>>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB