Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append


Copy link to this message
-
Re: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append
I think the key part is this:
java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting...
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680)
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)
> 2011-10-13 14:35:48,002 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0,
> memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31,
> maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440,
> blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9,
> blockCacheEvictedCount=0, blockCacheHitRatio=99,
> blockCacheHitCachingRatio=99
> 2011-10-13 14:35:48,002 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log
> roller
> 2011-10-13 14:35:48,002 INFO
> org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
>
>
This looks like the case that motivated HBASE-4222:
https://issues.apache.org/jira/browse/HBASE-4222

Did you by any chance restart the cluster DataNodes while the RegionServers
were running?  This can easily cause this issue to occur.  If not, look in
the DataNode logs for other errors which would indicate the underlying
problem.  In either case, the next release should be a bit more resilient to
this problem, as long as it is a transient error and you are not using
deferred log flushing on your tables.