Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Meta vs Stores / first and last keys and corrupted HFiles.


Copy link to this message
-
Re: Meta vs Stores / first and last keys and corrupted HFiles.
Matteo Bertozzi 2013-08-26, 19:37
time ago I started writing HBASE-7880.. that tries to rebuild an HFile
reading block per block without using the trailer... probably doesn't apply
cleanly but if you have time to play with it.. you may be able to extract
something from the hfile

Matteo

On Mon, Aug 26, 2013 at 8:31 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> Don't ask me how, but I have one table in a pretty strange state.
>
> First, seems that I have one corrupted HFile (at least).
>
> FirstKey return null. (Same for StopKey since header is corrupted).
>
> Exception in thread "main" java.lang.NullPointerException
>     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>     at
>
> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>     at
>
> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>
> Because the header is not correct.
>
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
> HFile Trailer from file
>
> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>     at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:551)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:595)
>     at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:642)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
> 16275367 (expected to be between 1 and 2)
>     at
> org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:771)
>     at
>
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:323)
>     at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:549)
>     ... 5 more
>
> HBCK don't detect that:
> 0 inconsistencies detected.
> Status: OK
>
> Also, I have an issue with the first key / last key reported by the stores
> overlapping when the one reported by the META are not.
>
> Store:
>     firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00...
>     lastKey=\xFF\xFF\xFF\xFE\x00\x00\x00\x00...
> Meta:
>     firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00
>     lastKey=\xF5\x9B@}\x00\x00\x00\x00...
>
>
> So, few things.
>
> 1) We should add something into HBCK to check the HFile format again
> corruption.
> 2) We should add something into HBCK to validate META regions boundaries
> against Store Files
> 3) How can I repaire my HFile? ;)
>
> I'm already working on #2 and will have something ready soon. Then I will
> most probably more to #1. But I only have detection dont for now. I'm not
> sure exactly what are the correct steps to repair...
>
> JM
>