Hi,
   the HMaster died as well as regionservers, below is hmaster's log. could
you please find what's problem?
2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181, initiating session
2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout = 40000
2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 15046ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 15245ms for sessionid
0x139c539bc090003, closing socket connection and attempting reconnect
2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, initiating session
2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: Unable to
reconnect to ZooKeeper service, session 0x139c539bc090003 has expired,
closing socket connection
2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, initiating session
2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn: Session
0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
    at sun.nio.ch.IOUtil.read(IOUtil.java:186)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
    at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181
2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 14007ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
10.20.16.32:2181
2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 26666ms for sessionid
0x139c539bc090002, closing socket connection and attempting reconnect
2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, initiating session
2012-10-12 00:14:46,726 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, sessionid = 0x139c539bc090002, negotiated timeout = 40000
2012-10-12 00:14:54,925 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 13464ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:14:56,524 ERROR org.apache.hadoop.hbase.master.HMaster:
Region server
serverName=bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2,60020,1347901025673,
load=(requests=75, regions=1, usedHeap=162, maxHeap=9725) reported a fatal
error:
ABORTING region server
serverName=bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2,60020,1347901025673,
load=(requests=75, regions=1, usedHeap=162, maxHeap=9725):
regionserver:60020-0x339c539ba640003 regionserver:60020-0x339c539ba640003
received expired from ZooKeeper, aborting
Cause:
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:353)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
    at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507)

2012-10-12 00:14:56,813 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:15:10,147 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 15119ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:15:10,625 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181
2012-10-12 00:15:10,625 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181, initiating session
2012-10-12 00:15:10,750 INFO org.apache.zookeeper.ClientCnxn: Unable to
reconnect to ZooKeeper service, session 0x239c539ba630001 has expired,
closing socket connection
2012-10-12 00:15:10,750 FATAL org.apache.hadoop.hbase.master.HMaster:
master:60000-0x239c539ba630001 master:60000-0x239c539ba6300

Check your GC configurations.  Seems to that a Full GC has happened and the
Zookeeper thought that to be session expiry.

Regards
Ram

> -----Original Message-----
> From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, October 13, 2012 6:20 PM
> To: [EMAIL PROTECTED]
> Subject: hmaster and regionserver died
>
> Hi,
>    the HMaster died as well as regionservers, below is hmaster's log.
> could
> you please find what's problem?
>
>
> 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> 10.20.16.34:2181, initiating session
> 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-
> 3/
> 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout > 40000
> 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 15046ms for sessionid
> 0x239c539ba630001, closing socket connection and attempting reconnect
> 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181
> 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 15245ms for sessionid
> 0x139c539bc090003, closing socket connection and attempting reconnect
> 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181
> 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181, initiating session
> 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> EventThread
> shut down
> 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x139c539bc090003 has expired,
> closing socket connection
> 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181, initiating session
> 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181, unexpected error, closing socket connection and
> attempting reconnect
> java.io.IOException: Connection reset by peer
>     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
>     at sun.nio.ch.IOUtil.read(IOUtil.java:186)
>     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
>     at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859)
>     at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
> 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> 10.20.16.34:2181
> 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 14007ms for sessionid
> 0x239c539ba630001, closing socket connection and attempting reconnect
> 2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
> 10.20.16.32:2181
> 2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 26666ms for sessionid
> 0x139c539bc090002, closing socket connection and attempting reconnect
> 2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181
> 2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/

We will check the zk log.

On Monday, October 15, 2012, Ramkrishna.S.Vasudevan wrote:

> Check your GC configurations.  Seems to that a Full GC has happened and the
> Zookeeper thought that to be session expiry.
>
> Regards
> Ram
>
> > -----Original Message-----
> > From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> > Sent: Saturday, October 13, 2012 6:20 PM
> > To: [EMAIL PROTECTED]
> > Subject: hmaster and regionserver died
> >
> > Hi,
> >    the HMaster died as well as regionservers, below is hmaster's log.
> > could
> > you please find what's problem?
> >
> >
> > 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn: Socket
> > connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> > 10.20.16.34:2181, initiating session
> > 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn: Session
> > establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-
> > 3/
> > 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout > > 40000
> > 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: Client
> > session timed out, have not heard from server in 15046ms for sessionid
> > 0x239c539ba630001, closing socket connection and attempting reconnect
> > 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn: Opening
> > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > 10.20.16.33:2181
> > 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn: Client
> > session timed out, have not heard from server in 15245ms for sessionid
> > 0x139c539bc090003, closing socket connection and attempting reconnect
> > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Opening
> > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > 10.20.16.33:2181
> > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Socket
> > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > 10.20.16.33:2181, initiating session
> > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> > EventThread
> > shut down
> > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: Unable to
> > reconnect to ZooKeeper service, session 0x139c539bc090003 has expired,
> > closing socket connection
> > 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn: Socket
> > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > 10.20.16.33:2181, initiating session
> > 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn: Session
> > 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > 10.20.16.33:2181, unexpected error, closing socket connection and
> > attempting reconnect
> > java.io.IOException: Connection reset by peer
> >     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> >     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> >     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
> >     at sun.nio.ch.IOUtil.read(IOUtil.java:186)
> >     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
> >     at
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859)
> >     at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
> > 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn: Opening
> > socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> > 10.20.16.34:2181
> > 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn: Client
> > session timed out, have not heard from server in 14007ms for sessionid
> > 0x239c539ba630001, closing socket connection and attempting reconnect
> > 2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn: Opening
> > socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
> > 10.20.16.32:2181
> > 2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn: Client
> > session timed out, have not heard from server in 26666ms for sessionid
>

Hi

Sorry if my reply mislead you.  I  meant to see the GC logs that should give
you an idea of if Full GC happened.

Regards
Ram

> -----Original Message-----
> From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> Sent: Monday, October 15, 2012 12:42 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hmaster and regionserver died
>
> We will check the zk log.
>
> On Monday, October 15, 2012, Ramkrishna.S.Vasudevan wrote:
>
> > Check your GC configurations.  Seems to that a Full GC has happened
> and the
> > Zookeeper thought that to be session expiry.
> >
> > Regards
> > Ram
> >
> > > -----Original Message-----
> > > From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> > > Sent: Saturday, October 13, 2012 6:20 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: hmaster and regionserver died
> > >
> > > Hi,
> > >    the HMaster died as well as regionservers, below is hmaster's
> log.
> > > could
> > > you please find what's problem?
> > >
> > >
> > > 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn:
> Socket
> > > connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> > > 10.20.16.34:2181, initiating session
> > > 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn:
> Session
> > > establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-
> stor-
> > > 3/
> > > 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout
> > > > 40000
> > > 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn:
> Client
> > > session timed out, have not heard from server in 15046ms for
> sessionid
> > > 0x239c539ba630001, closing socket connection and attempting
> reconnect
> > > 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn:
> Opening
> > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > 10.20.16.33:2181
> > > 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn:
> Client
> > > session timed out, have not heard from server in 15245ms for
> sessionid
> > > 0x139c539bc090003, closing socket connection and attempting
> reconnect
> > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn:
> Opening
> > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > 10.20.16.33:2181
> > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn:
> Socket
> > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > 10.20.16.33:2181, initiating session
> > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> > > EventThread
> > > shut down
> > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> Unable to
> > > reconnect to ZooKeeper service, session 0x139c539bc090003 has
> expired,
> > > closing socket connection
> > > 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn:
> Socket
> > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > 10.20.16.33:2181, initiating session
> > > 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn:
> Session
> > > 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-
> 2/
> > > 10.20.16.33:2181, unexpected error, closing socket connection and
> > > attempting reconnect
> > > java.io.IOException: Connection reset by peer
> > >     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> > >     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> > >     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
> > >     at sun.nio.ch.IOUtil.read(IOUtil.java:186)
> > >     at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
> > > 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn:
> Opening
> > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> > > 10.20.16.34:2181
> > > 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn:
> Client
> > > session timed out, have not heard from server in 14007ms for

HI,
   GC log is not opened. only zk's log, pls see below:
2012-10-12 00:14:30,470 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket
connection from /10.20.16.22:56954
2012-10-12 00:14:30,470 - ERROR [CommitProcessor:1:NIOServerCnxn@445] -
Unexpected Exception:
java.nio.channels.CancelledKeyException
 at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
 at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
 at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
 at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
 at
org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1545)
 at org.apache.zookeeper.server.DataTree.setWatches(DataTree.java:1156)
 at org.apache.zookeeper.server.ZKDatabase.setWatches(ZKDatabase.java:383)
 at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:267)
 at
org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
2012-10-12 00:14:30,470 - INFO  [QuorumPeer:/0.0.0.0:2181:NIOServerCnxn@1573]
- Invalid session 0x339c539ba64130b for client /10.20.16.29:41178, probably
expired
2012-10-12 00:14:30,470 - ERROR [CommitProcessor:1:NIOServerCnxn@445] -
Unexpected Exception:
java.nio.channels.CancelledKeyException
 at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
 at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
 at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
 at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
 at
org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1545)
 at org.apache.zookeeper.server.DataTree.setWatches(DataTree.java:1156)
 at org.apache.zookeeper.server.ZKDatabase.setWatches(ZKDatabase.java:383)
 at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:267)
 at
org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
2012-10-12 00:14:30,470 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@770] - Client attempting to renew
session 0x239c539ba632df9 at /10.20.16.23:37706
2012-10-12 00:14:30,470 - ERROR [CommitProcessor:1:NIOServerCnxn@445] -
Unexpected Exception:
java.nio.channels.CancelledKeyException
Best R.
beatls

On Mon, Oct 15, 2012 at 4:00 PM, Ramkrishna.S.Vasudevan <
[EMAIL PROTECTED]> wrote:

> Hi
>
> Sorry if my reply mislead you.  I  meant to see the GC logs that should
> give
> you an idea of if Full GC happened.
>
> Regards
> Ram
>
> > -----Original Message-----
> > From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, October 15, 2012 12:42 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: hmaster and regionserver died
> >
> > We will check the zk log.
> >
> > On Monday, October 15, 2012, Ramkrishna.S.Vasudevan wrote:
> >
> > > Check your GC configurations.  Seems to that a Full GC has happened
> > and the
> > > Zookeeper thought that to be session expiry.
> > >
> > > Regards
> > > Ram
> > >
> > > > -----Original Message-----
> > > > From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> > > > Sent: Saturday, October 13, 2012 6:20 PM
> > > > To: [EMAIL PROTECTED]
> > > > Subject: hmaster and regionserver died
> > > >
> > > > Hi,
> > > >    the HMaster died as well as regionservers, below is hmaster's
> > log.
> > > > could
> > > > you please find what's problem?
> > > >
> > > >
> > > > 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn:
> > Socket
> > > > connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> > > > 10.20.16.34:2181, initiating session
> > > > 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn:
> > Session
> > > > establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-
> > stor-
> > > > 3/
> > > > 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout
> > > > > > 40000
> > > > 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: