Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> zk keeps disconnecting and reconnecting


Copy link to this message
-
Re: zk keeps disconnecting and reconnecting
Ted,

Yes, we could have used just 3 servers. It just happens that we are
overlapping ZK servers on each of the 4 Kafka servers.

Jun

On Tue, Aug 23, 2011 at 4:07 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Jun,
>
> I don't see what your connection loss problem is, but I do have a question
> about why you are using 4 ZK servers.  This doesn't increase availability
> or
> durability over having just three ZK servers.
>
> On Tue, Aug 23, 2011 at 2:58 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > I have a ZK server cluster running on 4 nodes (version 3.3.3) and a few
> ZK
> > clients (version 3.3.0). After the clients have been running for a while,
> > each of them starts to constantly disconnect and reconnect to the ZK
> > server.
> > On the client, I saw lots of entries like the following:
> > 2011/08/23 14:42:06.579 INFO [ClientCnxn]
> > [main-SendThread(esv4-app27.stg:12913)] [kafka] Opening socket connection
> > to
> > server esv4-app28.stg/172.18.98.101:12913
> > 2011/08/23 14:42:06.579 INFO [ClientCnxn]
> > [main-SendThread(esv4-app28.stg:12913)] [kafka] Socket connection
> > established to esv4-app28.stg/172.18.98.101:12913, initiating session
> > 2011/08/23 14:42:06.581 INFO [ClientCnxn]
> > [main-SendThread(esv4-app28.stg:12913)] [kafka] Session establishment
> > complete on server esv4-app28.stg/172.18.98.101:12913, sessionid > > 0x331f77a1ed80004, negotiated timeout = 6000
> > 2011/08/23 14:42:06.581 INFO [ZkClient] [main-EventThread] [kafka]
> > zookeeper
> > state changed (SyncConnected)
> > 2011/08/23 14:42:06.583 WARN [ClientCnxn]
> > [main-SendThread(esv4-app28.stg:12913)] [kafka] Session 0x331f77a1ed80004
> > for server esv4-app28.stg/172.18.98.101:12913, unexpected error, closing
> > socket connection and attempting reconnect
> > java.lang.StringIndexOutOfBoundsException: String index out of range:
> > -3        at java.lang.String.substring(String.java:1937)
> >        at java.lang.String.substring(String.java:1904)
> >        at
> >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:753)
> > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:840)
> >        at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
> > 2011/08/23 14:42:06.683 INFO [ZkClient] [main-EventThread] [kafka]
> > zookeeper
> > state changed (Disconnected)
> > 2011/08/23 14:42:07.510 INFO [ClientCnxn]
> > [main-SendThread(esv4-app28.stg:12913)] [kafka] Opening socket connection
> > to
> > server esv4-app29.stg/172.18.98.89:12913
> > 2011/08/23 14:42:07.511 INFO [ClientCnxn]
> > [main-SendThread(esv4-app29.stg:12913)] [kafka] Socket connection
> > established to esv4-app29.stg/172.18.98.89:12913, initiating session
> > 2011/08/23 14:42:07.512 INFO [ClientCnxn]
> > [main-SendThread(esv4-app29.stg:12913)] [kafka] Session establishment
> > complete on server esv4-app29.stg/172.18.98.89:12913, sessionid = 0x331f
> > 77a1ed80004, negotiated timeout = 6000
> > 2011/08/23 14:42:07.513 INFO [ZkClient] [main-EventThread] [kafka]
> > zookeeper
> > state changed (SyncConnected)
> > 2011/08/23 14:42:07.552 WARN [ClientCnxn]
> > [main-SendThread(esv4-app29.stg:12913)] [kafka] Session 0x331f77a1ed80004
> > for server esv4-app29.stg/172.18.98.89:12913, unexpected error, clos
> > ing socket connection and attempting reconnect
> > java.lang.StringIndexOutOfBoundsException: String index out of range:
> > -3        at java.lang.String.substring(String.java:1937)
> >        at java.lang.String.substring(String.java:1904)
> >        at
> >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:753)
> > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:840)
> >        at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
> > 2011/08/23 14:42:07.653 INFO [ZkClient] [main-EventThread] [kafka]
> > zookeeper
> > state changed (Disconnected)
> >
> > On the ZK server, I saw lots of entries like these:
> > 2011-08-23 14:34:28,802 - INFO  [NIOServerCxn.Factory: