Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - Meta vs Stores / first and last keys and corrupted HFiles.


+
Jean-Marc Spaggiari 2013-08-26, 19:31
+
Kevin Odell 2013-08-26, 19:41
+
Jean-Marc Spaggiari 2013-08-26, 19:50
+
Matteo Bertozzi 2013-08-26, 19:37
+
Sergey Shelukhin 2013-08-26, 23:04
+
Jean-Marc Spaggiari 2013-08-26, 23:13
+
Jean-Marc Spaggiari 2013-08-26, 23:44
Copy link to this message
-
Re: Meta vs Stores / first and last keys and corrupted HFiles.
Jean-Marc Spaggiari 2013-08-27, 01:26
I have uploaded a patch in HBASE-9346 to add some check into HBCK. Working
pretty well.

To solve the issue, I "simply" merged the culprit region with the next one,
so there is no more issues with the region boundaries. Now I will major
compact and check the result. Then I will take a look at the NPE below...

2013/8/26 Jean-Marc Spaggiari <[EMAIL PROTECTED]>

> I have created HBASE-9346<https://issues.apache.org/jira/browse/HBASE-9346>for the boundaries issues.
>
>
> 2013/8/26 Jean-Marc Spaggiari <[EMAIL PROTECTED]>
>
>> You're right. I figured that after. So I removed the oldlogs (might be a
>> left over, was there for 2 days even after multiple stop/restart) but I'm
>> still getting this error on the region:
>>
>>
>> Exception in thread "main" java.lang.NullPointerException
>>     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>>     at
>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>>     at
>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>>     at
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>>     at
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>>     at
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>>     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>>
>> Even if now I'm able to get the keys information:
>>     firstKey=\x16\xCE\x8E\xCF\x00\x00\x00\x00...
>>     lastKey=\x19\xA0Ma\x00\x00\x00\x00...
>>
>> This is another region that the one where I have overlaps....
>>
>> JM
>>
>>
>> 2013/8/26 Sergey Shelukhin <[EMAIL PROTECTED]>
>>
>>> Um, I may be missing something here, but is it really an HFile?
>>> Looks like an hlog to me.
>>>
>>> org.apache.hadoop.hbase.io.
>>> hfile.CorruptHFileException: Problem reading
>>> HFile Trailer from file
>>>
>>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>>>
>>>
>>> On Mon, Aug 26, 2013 at 12:31 PM, Jean-Marc Spaggiari <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>> > Hi,
>>> >
>>> > Don't ask me how, but I have one table in a pretty strange state.
>>> >
>>> > First, seems that I have one corrupted HFile (at least).
>>> >
>>> > FirstKey return null. (Same for StopKey since header is corrupted).
>>> >
>>> > Exception in thread "main" java.lang.NullPointerException
>>> >     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>>> >     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>>> >
>>> > Because the header is not correct.
>>> >
>>> > org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
>>> > HFile Trailer from file
>>> >
>>> >
>>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>>> >     at
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:551)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:595)
>>> >     at
>>> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:642)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)