Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> zkclient timeout issue


Copy link to this message
-
Re: zkclient timeout issue
Are you sure that the JVM is not swapping? Perhaps the "disk intensive
process" is also eating up alot of memory and that's pushing ZK server
out to swap?

Turn swappyness off and retry.

Patrick

On Thu, Oct 10, 2013 at 10:31 AM, Srikanth R <[EMAIL PROTECTED]> wrote:
> Hi Guys,
>
> Need some expert advice :)
>
> I have a 3 Server Zookeeper ensemble. Only one client (hadoop-zkfc with
> 5second timeout)  is connected to zookeeper. There is no other activity,
> one txn is recorded every 10 min in zookeeper txn log. Zookeeper is as
> close to being idle, except for the session traffic from the hadoop-zkfc
> client.
>
> Even with no writes happening to the zookeeper data dir, if  I start some
> disk intensive process on the same partition  that has datadir (like
> raid-check or cat'ing huge files), I am observing zkclient session
> timeouts. (Error: Client has not heard back from Server in 3334 ms, so
> disconnecting and reconnecting)
>
> strace on zookeeper process shows that it received the 12byte heartbeat
> from the client, but has not responded. Also strace does not show any disk
> activity from zoookeper.
>
> So the question is does zookeeper do anything on the disk even when its
> idle (not writing any txns to disk). why does the datadir disk utilization
> affect zookeeper even without any traffic ?
>
> Any help is appreciated in this regard.
>
> Thanks,
> Srikanth
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB