Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Meta vs Stores / first and last keys and corrupted HFiles.


+
Jean-Marc Spaggiari 2013-08-26, 19:31
+
Kevin Odell 2013-08-26, 19:41
+
Jean-Marc Spaggiari 2013-08-26, 19:50
+
Matteo Bertozzi 2013-08-26, 19:37
+
Sergey Shelukhin 2013-08-26, 23:04
+
Jean-Marc Spaggiari 2013-08-26, 23:13
+
Jean-Marc Spaggiari 2013-08-26, 23:44
Copy link to this message
-
Re: Meta vs Stores / first and last keys and corrupted HFiles.
I have uploaded a patch in HBASE-9346 to add some check into HBCK. Working
pretty well.

To solve the issue, I "simply" merged the culprit region with the next one,
so there is no more issues with the region boundaries. Now I will major
compact and check the result. Then I will take a look at the NPE below...

2013/8/26 Jean-Marc Spaggiari <[EMAIL PROTECTED]>

> I have created HBASE-9346<https://issues.apache.org/jira/browse/HBASE-9346>for the boundaries issues.
>
>
> 2013/8/26 Jean-Marc Spaggiari <[EMAIL PROTECTED]>
>
>> You're right. I figured that after. So I removed the oldlogs (might be a
>> left over, was there for 2 days even after multiple stop/restart) but I'm
>> still getting this error on the region:
>>
>>
>> Exception in thread "main" java.lang.NullPointerException
>>     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>>     at
>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>>     at
>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>>     at
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>>     at
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>>     at
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>>     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>>
>> Even if now I'm able to get the keys information:
>>     firstKey=\x16\xCE\x8E\xCF\x00\x00\x00\x00...
>>     lastKey=\x19\xA0Ma\x00\x00\x00\x00...
>>
>> This is another region that the one where I have overlaps....
>>
>> JM
>>
>>
>> 2013/8/26 Sergey Shelukhin <[EMAIL PROTECTED]>
>>
>>> Um, I may be missing something here, but is it really an HFile?
>>> Looks like an hlog to me.
>>>
>>> org.apache.hadoop.hbase.io.
>>> hfile.CorruptHFileException: Problem reading
>>> HFile Trailer from file
>>>
>>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>>>
>>>
>>> On Mon, Aug 26, 2013 at 12:31 PM, Jean-Marc Spaggiari <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>> > Hi,
>>> >
>>> > Don't ask me how, but I have one table in a pretty strange state.
>>> >
>>> > First, seems that I have one corrupted HFile (at least).
>>> >
>>> > FirstKey return null. (Same for StopKey since header is corrupted).
>>> >
>>> > Exception in thread "main" java.lang.NullPointerException
>>> >     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>>> >     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>>> >
>>> > Because the header is not correct.
>>> >
>>> > org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
>>> > HFile Trailer from file
>>> >
>>> >
>>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>>> >     at
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:551)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:595)
>>> >     at
>>> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:642)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB