Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Zookeeper issue? (ConnectionLoss for /hbase/hbaseid)


Copy link to this message
-
Zookeeper issue? (ConnectionLoss for /hbase/hbaseid)
Hi,

I ran a 30h MapReduce job, and now I'm not able to connect anymore to
my HBase cluster.

The MapReduce was configured on ReadOnly mode. So only the log table
received data. Everything else was just ready.

Today I killed the job to replace one of the servers which is too slow
and now I'm not able to connect to ZooKeeper anymore.

Below is the stack trace, and at the bottom is the ZKDump.

I think if I restart everything it should be working, but I'm
wondering if there is any information on this situation which might
help to prevent this to happend in the futur? I might be able to
reproduce that since I can re-run the job almost anytime.

I don't seems to have to many connections. The HBase shell is replying
correctly. It's really only the Java application using ZooKeeper which
is not working.

I'm running HBase 0.94.2, ZK 3.4.3, Hadoop 1.0.3 all installed separatly.

JM

2012-11-15 18:09:33,684 [main-SendThread(cube:21818)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connexion refuse
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2012-11-15 18:09:33,810 [main] WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
2012-11-15 18:09:34,800 [main-SendThread(cube:21818)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connexion refuse
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2012-11-15 18:09:35,902 [main-SendThread(cube:21818)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connexion refuse
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2012-11-15 18:09:36,003 [main] WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
2012-11-15 18:09:37,004 [main-SendThread(cube:21818)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connexion refuse
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2012-11-15 18:09:38,106 [main-SendThread(cube:21818)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connexion refuse
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2012-11-15 18:09:39,209 [main-SendThread(cube:21818)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connexion refuse
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2012-11-15 18:09:40,311 [main-SendThread(cube:21818)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connexion refuse
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2012-11-15 18:09:40,412 [main] WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
2012-11-15 18:09:41,414 [main-SendThread(cube:21818)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connexion refuse
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2012-11-15 18:09:42,516 [main-SendThread(cube:21818)] WARN
org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,