Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - hmaster and regionserver died


Copy link to this message
-
RE: hmaster and regionserver died
Ramkrishna.S.Vasudevan 2012-10-15, 04:36
Check your GC configurations.  Seems to that a Full GC has happened and the
Zookeeper thought that to be session expiry.

Regards
Ram

> -----Original Message-----
> From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, October 13, 2012 6:20 PM
> To: [EMAIL PROTECTED]
> Subject: hmaster and regionserver died
>
> Hi,
>    the HMaster died as well as regionservers, below is hmaster's log.
> could
> you please find what's problem?
>
>
> 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> 10.20.16.34:2181, initiating session
> 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-
> 3/
> 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout > 40000
> 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 15046ms for sessionid
> 0x239c539ba630001, closing socket connection and attempting reconnect
> 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181
> 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 15245ms for sessionid
> 0x139c539bc090003, closing socket connection and attempting reconnect
> 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181
> 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181, initiating session
> 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> EventThread
> shut down
> 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x139c539bc090003 has expired,
> closing socket connection
> 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181, initiating session
> 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181, unexpected error, closing socket connection and
> attempting reconnect
> java.io.IOException: Connection reset by peer
>     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
>     at sun.nio.ch.IOUtil.read(IOUtil.java:186)
>     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
>     at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859)
>     at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
> 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> 10.20.16.34:2181
> 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 14007ms for sessionid
> 0x239c539ba630001, closing socket connection and attempting reconnect
> 2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
> 10.20.16.32:2181
> 2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 26666ms for sessionid
> 0x139c539bc090002, closing socket connection and attempting reconnect
> 2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181
> 2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/