-Re: corrupt WAL and Java Heap Space...
Dave Latham 2011-08-26, 16:44
We just hit the same issue. I attached log snippets from the regionserver
and master into https://issues.apache.org/jira/browse/HBASE-4107
I was able to get the log file out of hdfs. Is there a location I can put
it back in to have it picked up?
On Fri, Jul 15, 2011 at 12:23 PM, Andy Sautins
> I don't have the log still. Not sure what I was thinking deleting it. I
> was a little too aggressive wanting to get my fsck back to having 0 corrupt
> What you say is interesting. It's more than possible that I'm
> misunderstanding what is going on.
> What we saw with the log file is that we could cat it, but couldn't copy
> the file ( would complain about a bad checksum ). I know that's not hard
> data, but going by that what you say about applying the log up until the
> last sync makes would make sense. What might have thrown me is after a
> re-start the logs ( including the corrupt log ) were still in the .logs
> folder. We did a full shutdown/restart and the following stacktrace was in
> the master logs. After this stacktrace hbase continued to startup, however
> the logs ( all logs up until the corrupt log ) for the region with the
> corrupt log file were left in the .logs directory. When we removed the
> corrupt log file and re-started again all the existing logs were removed
> after successful restart as I would expect.
> So is it more likely that the error on shutdown is reasonable and that
> the log cleanup just didn't happen on startup? I suppose it makes sense not
> to remove them if there is an error, but it did throw me that the corrupt
> file as well as previous files were still in the .logs directory.
> 2011-07-14 18:07:45,954 ERROR
> org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting hdfs://
> org.apache.hadoop.fs.ChecksumException: Checksum error:
> ,60020,1309294522164/hd31.dfs.returnpath.net%3A60020.1310675410770 at
> at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)