Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> zk keeps disconnecting and reconnecting


+
Jun Rao 2011-08-23, 21:58
+
Ted Dunning 2011-08-23, 23:07
+
Jun Rao 2011-08-23, 23:24
+
Mahadev Konar 2011-08-24, 03:42
+
Patrick Hunt 2011-08-25, 16:34
Copy link to this message
-
Re: zk keeps disconnecting and reconnecting
We cleaned up all ZK server data and restarted both the servers and the
clients. We also upgraded the client to 3.3.3. After running for a day and a
half, the same weird reconnect issue showed up in one of the clients. Our ZK
connection string is
esv4-app27.stg:12913,esv4-app28.stg:12913,esv4-app29.stg:12913,esv4-app30.stg:12913/kafka-tracking.
We are on java 1_6_0_21 on RedHat Linux. Note that our ZK client has been
running fine until we upgraded the client code recently. The new version
makes one extra ZK connection to the same ZK cluster. Here are the log
entries and ZK client keeps connecting and disconnecting from each of the 4
ZK servers.

2011/08/26 10:58:39.864 INFO [ClientCnxn]
[main-SendThread(esv4-app28.stg:12913)] [kafka] Opening socket connection to
server esv4-app27.stg/172.18.98.88:12913
2011/08/26 10:58:39.865 INFO [ClientCnxn]
[main-SendThread(esv4-app27.stg:12913)] [kafka] Socket connection
established to esv4-app27.stg/172.18.98.88:12913, initiating session
2011/08/26 10:58:39.867 INFO [ClientCnxn]
[main-SendThread(esv4-app27.stg:12913)] [kafka] Session establishment
complete on server esv4-app27.stg/172.18.98.88:12913, sessionid 0x131fddd84bc0006, negotiated timeout = 6000
2011/08/26 10:58:39.867 INFO [ZkClient] [main-EventThread] [kafka] zookeeper
state changed (SyncConnected)
2011/08/26 10:58:39.868 WARN [ClientCnxn]
[main-SendThread(esv4-app27.stg:12913)] [kafka] Session 0x131fddd84bc0006
for server esv4-app27.stg/172.18.98.88:12913, unexpected error, closing
socket connection and attempting reconnect
java.lang.StringIndexOutOfBoundsException: String index out of range: -3
        at java.lang.String.substring(String.java:1937)
        at java.lang.String.substring(String.java:1904)
        at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:794)
        at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:881)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
2011/08/26 10:58:39.969 INFO [ZkClient] [main-EventThread] [kafka] zookeeper
state changed (Disconnected)
2011/08/26 10:58:40.276 INFO [ClientCnxn]
[main-SendThread(esv4-app27.stg:12913)] [kafka] Opening socket connection to
server esv4-app29.stg/172.18.98.89:12913
2011/08/26 10:58:40.276 INFO [ClientCnxn]
[main-SendThread(esv4-app29.stg:12913)] [kafka] Socket connection
established to esv4-app29.stg/172.18.98.89:12913, initiating session
2011/08/26 10:58:40.278 INFO [ClientCnxn]
[main-SendThread(esv4-app29.stg:12913)] [kafka] Session establishment
complete on server esv4-app29.stg/172.18.98.89:12913, sessionid 0x131fddd84bc0006, negotiated timeout = 6000
2011/08/26 10:58:40.278 INFO [ZkClient] [main-EventThread] [kafka] zookeeper
state changed (SyncConnected)
2011/08/26 10:58:40.279 WARN [ClientCnxn]
[main-SendThread(esv4-app29.stg:12913)] [kafka] Session 0x131fddd84bc0006
for server esv4-app29.stg/172.18.98.89:12913, unexpected error, closing
socket connection and attempting reconnect
java.lang.StringIndexOutOfBoundsException: String index out of range: -3
        at java.lang.String.substring(String.java:1937)
        at java.lang.String.substring(String.java:1904)
        at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:794)
        at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:881)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
2011/08/26 10:58:40.380 INFO [ZkClient] [main-EventThread] [kafka] zookeeper
state changed (Disconnected)
2011/08/26 10:58:40.515 INFO [ClientCnxn]
[main-SendThread(esv4-app29.stg:12913)] [kafka] Opening socket connection to
server esv4-app30.stg/172.18.98.90:12913

Thanks,

Jun

On Thu, Aug 25, 2011 at 9:34 AM, Patrick Hunt <[EMAIL PROTECTED]> wrote:

> The client seeing the problem in this case is 3.3.0, I see this based
> on the line number in the stack trace not matching up with 3.3.3, with
> 3.3.0 it's this line:
>
>           event.setPath(serverPath.substring(chrootPath.length()));
+
Jun Rao 2011-08-29, 16:39
+
Fournier, Camille F. 2011-08-29, 17:50
+
Mahadev Konar 2011-08-29, 18:10
+
Fournier, Camille F. 2011-08-29, 19:38
+
Mahadev Konar 2011-08-29, 19:45
+
Benjamin Reed 2011-08-30, 23:45
+
Camille Fournier 2011-08-30, 23:50
+
Jun Rao 2011-08-31, 01:48
+
kishore g 2011-08-31, 17:02
+
Fournier, Camille F. 2011-09-01, 14:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB