Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> hmaster and regionserver died


Copy link to this message
-
RE: hmaster and regionserver died
Hi

Sorry if my reply mislead you.  I  meant to see the GC logs that should give
you an idea of if Full GC happened.

Regards
Ram

> -----Original Message-----
> From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> Sent: Monday, October 15, 2012 12:42 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hmaster and regionserver died
>
> We will check the zk log.
>
> On Monday, October 15, 2012, Ramkrishna.S.Vasudevan wrote:
>
> > Check your GC configurations.  Seems to that a Full GC has happened
> and the
> > Zookeeper thought that to be session expiry.
> >
> > Regards
> > Ram
> >
> > > -----Original Message-----
> > > From: Xiang Hua [mailto:[EMAIL PROTECTED]]
> > > Sent: Saturday, October 13, 2012 6:20 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: hmaster and regionserver died
> > >
> > > Hi,
> > >    the HMaster died as well as regionservers, below is hmaster's
> log.
> > > could
> > > you please find what's problem?
> > >
> > >
> > > 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn:
> Socket
> > > connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> > > 10.20.16.34:2181, initiating session
> > > 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn:
> Session
> > > establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-
> stor-
> > > 3/
> > > 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout
> > > > 40000
> > > 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn:
> Client
> > > session timed out, have not heard from server in 15046ms for
> sessionid
> > > 0x239c539ba630001, closing socket connection and attempting
> reconnect
> > > 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn:
> Opening
> > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > 10.20.16.33:2181
> > > 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn:
> Client
> > > session timed out, have not heard from server in 15245ms for
> sessionid
> > > 0x139c539bc090003, closing socket connection and attempting
> reconnect
> > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn:
> Opening
> > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > 10.20.16.33:2181
> > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn:
> Socket
> > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > 10.20.16.33:2181, initiating session
> > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> > > EventThread
> > > shut down
> > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn:
> Unable to
> > > reconnect to ZooKeeper service, session 0x139c539bc090003 has
> expired,
> > > closing socket connection
> > > 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn:
> Socket
> > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
> > > 10.20.16.33:2181, initiating session
> > > 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn:
> Session
> > > 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-
> 2/
> > > 10.20.16.33:2181, unexpected error, closing socket connection and
> > > attempting reconnect
> > > java.io.IOException: Connection reset by peer
> > >     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> > >     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> > >     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
> > >     at sun.nio.ch.IOUtil.read(IOUtil.java:186)
> > >     at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
> > > 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn:
> Opening
> > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
> > > 10.20.16.34:2181
> > > 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn:
> Client
> > > session timed out, have not heard from server in 14007ms for