Hi Friends
I am using a mini distributed cluster of 3 nodes. Regionservers are running on all of 3 nodes. This cluster is operational from 2 weeks now without any issue. Today morning I saw one of the region servers are down.
I restarted the regionserver dynamically and it started fine and master also distributed regions to this region server.
I looked at the log but not much understood what was the problem. Logs state that one of my datanodes are bad, but when I checked, all datanodes are working fine.

Please suggest

Regionserver Log :

2011-10-13 14:35:47,974 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog
java.io.IOException: Reflection
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1477)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2549)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
        ... 10 more
Caused by: java.io.IOException: DFSOutputStream is closed
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238)
        at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
        at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
        ... 14 more
2011-10-13 14:35:47,999 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2011-10-13 14:35:48,001 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=master,60020,1316669478524, load=(requests=0, regions=4, usedHeap=31, maxHeap=993): IOE in log roller
java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0, memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31, maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440, blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9, blockCacheEvictedCount=0, blockCacheHitRatio=99, blockCacheHitCachingRatio=99
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log roller
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.

::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of
this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have
received this email in error please delete it and notify the sender immediately. Before opening any mail and
attachments please check them for viruses and defect.

Did you do some heavy inserting during that time? I assume you use 0.90.x(?)

Might be HBASE-4387.

-- Lars

----- Original Message -----
From: Stuti Awasthi <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc:
Sent: Thursday, October 13, 2011 9:44 PM
Subject: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append

Hi Friends
I am using a mini distributed cluster of 3 nodes. Regionservers are running on all of 3 nodes. This cluster is operational from 2 weeks now without any issue. Today morning I saw one of the region servers are down.
I restarted the regionserver dynamically and it started fine and master also distributed regions to this region server.
I looked at the log but not much understood what was the problem. Logs state that one of my datanodes are bad, but when I checked, all datanodes are working fine.

Please suggest

Regionserver Log :

2011-10-13 14:35:47,974 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog
java.io.IOException: Reflection
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1477)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2549)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
        ... 10 more
Caused by: java.io.IOException: DFSOutputStream is closed
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238)
        at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
        at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
        ... 14 more
2011-10-13 14:35:47,999 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2011-10-13 14:35:48,001 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=master,60020,1316669478524, load=(requests=0, regions=4, usedHeap=31, maxHeap=993): IOE in log roller
java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0, memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31, maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440, blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9, blockCacheEvictedCount=0, blockCacheHitRatio=99, blockCacheHitCachingRatio=99
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log roller
2011-10-13 14:35

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of
this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have
received this email in error please delete it and notify the sender immediately. Before opening any mail and
attachments please check them for viruses and defect.

Hi Lars,
I am using Hbase 0.90.3 and there was no inserting done on the nodes at that time.

-----Original Message-----
From: lars hofhansl [mailto:[EMAIL PROTECTED]]
Sent: Friday, October 14, 2011 10:46 AM
To: [EMAIL PROTECTED]
Subject: Re: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append

Did you do some heavy inserting during that time? I assume you use 0.90.x(?)

Might be HBASE-4387.

-- Lars

----- Original Message -----
From: Stuti Awasthi <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc:
Sent: Thursday, October 13, 2011 9:44 PM
Subject: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append

Hi Friends
I am using a mini distributed cluster of 3 nodes. Regionservers are running on all of 3 nodes. This cluster is operational from 2 weeks now without any issue. Today morning I saw one of the region servers are down.
I restarted the regionserver dynamically and it started fine and master also distributed regions to this region server.
I looked at the log but not much understood what was the problem. Logs state that one of my datanodes are bad, but when I checked, all datanodes are working fine.

Please suggest

Regionserver Log :

2011-10-13 14:35:47,974 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog
java.io.IOException: Reflection
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1477)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2549)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
        ... 10 more
Caused by: java.io.IOException: DFSOutputStream is closed
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238)
        at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
        at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
        ... 14 more
2011-10-13 14:35:47,999 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2011-10-13 14:35:48,001 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=master,60020,1316669478524, load=(requests=0, regions=4, usedHeap=31, maxHeap=993): IOE in log roller
java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0, memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31, maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440, blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9, blockCacheEvictedCount=0, blockCacheHitRatio=99, blockCacheHitCachingRatio=99
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log roller
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any mail and attachments please check them for viruses and defect.

Hi

Are you sure there was no insert.  The stack trace shows a put operation was
going on.  You can check your datanode logs at the corresponding time when
this exception occured in the RS to see if any write request has reached the
DN and that you can find using the HLog file name for that RS.

Regards
Ram

-----Original Message-----
From: Stuti Awasthi [mailto:[EMAIL PROTECTED]]
Sent: Friday, October 14, 2011 2:47 PM
To: [EMAIL PROTECTED]
Subject: RE: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not
append

Hi Lars,
I am using Hbase 0.90.3 and there was no inserting done on the nodes at that
time.

-----Original Message-----
From: lars hofhansl [mailto:[EMAIL PROTECTED]]
Sent: Friday, October 14, 2011 10:46 AM
To: [EMAIL PROTECTED]
Subject: Re: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not
append

Did you do some heavy inserting during that time? I assume you use 0.90.x(?)

Might be HBASE-4387.

-- Lars

----- Original Message -----
From: Stuti Awasthi <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc:
Sent: Thursday, October 13, 2011 9:44 PM
Subject: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not
append

Hi Friends
I am using a mini distributed cluster of 3 nodes. Regionservers are running
on all of 3 nodes. This cluster is operational from 2 weeks now without any
issue. Today morning I saw one of the region servers are down.
I restarted the regionserver dynamically and it started fine and master also
distributed regions to this region server.
I looked at the log but not much understood what was the problem. Logs state
that one of my datanodes are bad, but when I checked, all datanodes are
working fine.

Please suggest

Regionserver Log :

2011-10-13 14:35:47,974 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog:
Could not append. Requesting close of hlog
java.io.IOException: Reflection
        at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(Sequence
FileLogWriter.java:147)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
        at
org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937)
        at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:147
7)
        at
org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2549)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(Sequence
FileLogWriter.java:145)
        ... 10 more
Caused by: java.io.IOException: DFSOutputStream is closed
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238)
        at
org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
        at
org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
        ... 14 more
2011-10-13 14:35:47,999 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
-- HDFS-200
2011-10-13 14:35:48,001 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
serverName=master,60020,1316669478524, load=(requests=0, regions=4,
usedHeap=31, maxHeap=993): IOE in log roller
java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting...
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSCli
ent.java:2680)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:
2172)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.
java:2371)
2011-10-13 14:35:48,002 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0,
memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31,
maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440,
blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9,
blockCacheEvictedCount=0, blockCacheHitRatio=99,
blockCacheHitCachingRatio=99
2011-10-13 14:35:48,002 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log
roller
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
The contents of this e-mail and any attachment(s) are confidential and
intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its
affiliates. Any views or opinions presented in this email are solely those
of the author and may not necessarily reflect the opinions of HCL or its
affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written
consent of the author of this e-mail is strictly prohibited. If you have
received this email in error please delete it and notify the sender
immediately. Before opening any mail and attachments please check them for
viruses and defect.

Also make sure this isn't an exception that came after a bunch of others,
look for the first FATAL.

J-D

On Fri, Oct 14, 2011 at 2:25 AM, Ramkrishna S Vasudevan <
[EMAIL PROTECTED]> wrote:

> Hi
>
> Are you sure there was no insert.  The stack trace shows a put operation
> was
> going on.  You can check your datanode logs at the corresponding time when
> this exception occured in the RS to see if any write request has reached
> the
> DN and that you can find using the HLog file name for that RS.
>
> Regards
> Ram
>
> -----Original Message-----
> From: Stuti Awasthi [mailto:[EMAIL PROTECTED]]
> Sent: Friday, October 14, 2011 2:47 PM
> To: [EMAIL PROTECTED]
> Subject: RE: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not
> append
>
> Hi Lars,
> I am using Hbase 0.90.3 and there was no inserting done on the nodes at
> that
> time.
>
> -----Original Message-----
> From: lars hofhansl [mailto:[EMAIL PROTECTED]]
> Sent: Friday, October 14, 2011 10:46 AM
> To: [EMAIL PROTECTED]
> Subject: Re: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not
> append
>
> Did you do some heavy inserting during that time? I assume you use
> 0.90.x(?)
>
> Might be HBASE-4387.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Stuti Awasthi <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Cc:
> Sent: Thursday, October 13, 2011 9:44 PM
> Subject: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not
> append
>
> Hi Friends
> I am using a mini distributed cluster of 3 nodes. Regionservers are running
> on all of 3 nodes. This cluster is operational from 2 weeks now without any
> issue. Today morning I saw one of the region servers are down.
> I restarted the regionserver dynamically and it started fine and master
> also
> distributed regions to this region server.
> I looked at the log but not much understood what was the problem. Logs
> state
> that one of my datanodes are bad, but when I checked, all datanodes are
> working fine.
>
> Please suggest
>
> Regionserver Log :
>
> 2011-10-13 14:35:47,974 FATAL
> org.apache.hadoop.hbase.regionserver.wal.HLog:
> Could not append. Requesting close of hlog
> java.io.IOException: Reflection
>         at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(Sequence
> FileLogWriter.java:147)
>         at
> org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
>         at
> org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:147
> 7)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2549)
>         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(Sequence
> FileLogWriter.java:145)
>         ... 10 more
> Caused by: java.io.IOException: DFSOutputStream is closed
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238)
>         at
> org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
>         at
> org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
>         ... 14 more

I think the key part is this:
java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting...
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680)
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)
> 2011-10-13 14:35:48,002 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0,
> memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31,
> maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440,
> blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9,
> blockCacheEvictedCount=0, blockCacheHitRatio=99,
> blockCacheHitCachingRatio=99
> 2011-10-13 14:35:48,002 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log
> roller
> 2011-10-13 14:35:48,002 INFO
> org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
>
>
This looks like the case that motivated HBASE-4222:
https://issues.apache.org/jira/browse/HBASE-4222

Did you by any chance restart the cluster DataNodes while the RegionServers
were running?  This can easily cause this issue to occur.  If not, look in
the DataNode logs for other errors which would indicate the underlying
problem.  In either case, the next release should be a bit more resilient to
this problem, as long as it is a transient error and you are not using
deferred log flushing on your tables.