Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> zk keeps disconnecting and reconnecting


Copy link to this message
-
Re: zk keeps disconnecting and reconnecting
Ted,

Yes, we could have used just 3 servers. It just happens that we are
overlapping ZK servers on each of the 4 Kafka servers.

Jun

On Tue, Aug 23, 2011 at 4:07 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Jun,
>
> I don't see what your connection loss problem is, but I do have a question
> about why you are using 4 ZK servers.  This doesn't increase availability
> or
> durability over having just three ZK servers.
>
> On Tue, Aug 23, 2011 at 2:58 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > I have a ZK server cluster running on 4 nodes (version 3.3.3) and a few
> ZK
> > clients (version 3.3.0). After the clients have been running for a while,
> > each of them starts to constantly disconnect and reconnect to the ZK
> > server.
> > On the client, I saw lots of entries like the following:
> > 2011/08/23 14:42:06.579 INFO [ClientCnxn]
> > [main-SendThread(esv4-app27.stg:12913)] [kafka] Opening socket connection
> > to
> > server esv4-app28.stg/172.18.98.101:12913
> > 2011/08/23 14:42:06.579 INFO [ClientCnxn]
> > [main-SendThread(esv4-app28.stg:12913)] [kafka] Socket connection
> > established to esv4-app28.stg/172.18.98.101:12913, initiating session
> > 2011/08/23 14:42:06.581 INFO [ClientCnxn]
> > [main-SendThread(esv4-app28.stg:12913)] [kafka] Session establishment
> > complete on server esv4-app28.stg/172.18.98.101:12913, sessionid > > 0x331f77a1ed80004, negotiated timeout = 6000
> > 2011/08/23 14:42:06.581 INFO [ZkClient] [main-EventThread] [kafka]
> > zookeeper
> > state changed (SyncConnected)
> > 2011/08/23 14:42:06.583 WARN [ClientCnxn]
> > [main-SendThread(esv4-app28.stg:12913)] [kafka] Session 0x331f77a1ed80004
> > for server esv4-app28.stg/172.18.98.101:12913, unexpected error, closing
> > socket connection and attempting reconnect
> > java.lang.StringIndexOutOfBoundsException: String index out of range:
> > -3        at java.lang.String.substring(String.java:1937)
> >        at java.lang.String.substring(String.java:1904)
> >        at
> >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:753)
> > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:840)
> >        at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
> > 2011/08/23 14:42:06.683 INFO [ZkClient] [main-EventThread] [kafka]
> > zookeeper
> > state changed (Disconnected)
> > 2011/08/23 14:42:07.510 INFO [ClientCnxn]
> > [main-SendThread(esv4-app28.stg:12913)] [kafka] Opening socket connection
> > to
> > server esv4-app29.stg/172.18.98.89:12913
> > 2011/08/23 14:42:07.511 INFO [ClientCnxn]
> > [main-SendThread(esv4-app29.stg:12913)] [kafka] Socket connection
> > established to esv4-app29.stg/172.18.98.89:12913, initiating session
> > 2011/08/23 14:42:07.512 INFO [ClientCnxn]
> > [main-SendThread(esv4-app29.stg:12913)] [kafka] Session establishment
> > complete on server esv4-app29.stg/172.18.98.89:12913, sessionid = 0x331f
> > 77a1ed80004, negotiated timeout = 6000
> > 2011/08/23 14:42:07.513 INFO [ZkClient] [main-EventThread] [kafka]
> > zookeeper
> > state changed (SyncConnected)
> > 2011/08/23 14:42:07.552 WARN [ClientCnxn]
> > [main-SendThread(esv4-app29.stg:12913)] [kafka] Session 0x331f77a1ed80004
> > for server esv4-app29.stg/172.18.98.89:12913, unexpected error, clos
> > ing socket connection and attempting reconnect
> > java.lang.StringIndexOutOfBoundsException: String index out of range:
> > -3        at java.lang.String.substring(String.java:1937)
> >        at java.lang.String.substring(String.java:1904)
> >        at
> >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:753)
> > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:840)
> >        at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
> > 2011/08/23 14:42:07.653 INFO [ZkClient] [main-EventThread] [kafka]
> > zookeeper
> > state changed (Disconnected)
> >
> > On the ZK server, I saw lots of entries like these:
> > 2011-08-23 14:34:28,802 - INFO  [NIOServerCxn.Factory:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB