Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> hmaster and regionserver died


Copy link to this message
-
RE: hmaster and regionserver died
Check your GC configurations.  Seems to that a Full GC has happened and the
Zookeeper thought that to be session expiry.

Regards
Ram

> -----Original Message-----
> From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, October 13, 2012 6:20 PM
> To: [EMAIL PROTECTED]
> Subject: hmaster and regionserver died
>
> Hi,
>    the HMaster died as well as regionservers, below is hmaster's log.
> could
> you please find what's problem?
>
>
> 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> 10.20.16.34:2181, initiating session
> 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-
> 3/
> 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout > 40000
> 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 15046ms for sessionid
> 0x239c539ba630001, closing socket connection and attempting reconnect
> 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181
> 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 15245ms for sessionid
> 0x139c539bc090003, closing socket connection and attempting reconnect
> 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181
> 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181, initiating session
> 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> EventThread
> shut down
> 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x139c539bc090003 has expired,
> closing socket connection
> 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181, initiating session
> 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181, unexpected error, closing socket connection and
> attempting reconnect
> java.io.IOException: Connection reset by peer
>     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
>     at sun.nio.ch.IOUtil.read(IOUtil.java:186)
>     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
>     at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859)
>     at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
> 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> 10.20.16.34:2181
> 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 14007ms for sessionid
> 0x239c539ba630001, closing socket connection and attempting reconnect
> 2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
> 10.20.16.32:2181
> 2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 26666ms for sessionid
> 0x139c539bc090002, closing socket connection and attempting reconnect
> 2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> 10.20.16.33:2181
> 2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB