Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> determining ZK latency issues


Copy link to this message
-
Re: determining ZK latency issues
Naha,

Do you have the distribution of the request latencies? If you're seeing
something like the top 1% of requests taking 5s while the rest take double
digit ms, and average out to three-digit ms... then it's likely that some
requests are still waiting on the snapshot write despite you having
disabled wait for fsync.

If the distribution is not spiky though - then I haven't seen that case.

Best Regards,
Martin Kou
On Thu, Aug 8, 2013 at 11:40 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Since this zookeeper cluster is a pre-prod cluster and is used heavily, we
> have turned the forceSync to OFF. So we don't see the fsync warnings on
> this zookeeper cluster.
>
> Thanks,
> Neha
>
>
> On Thu, Aug 8, 2013 at 11:18 AM, Martin Kou <[EMAIL PROTECTED]> wrote:
>
> > Do you see fsync() warnings in your ZooKeeper log? It could be disk
> writes
> > taking too long.
> >
> > Best Regards,
> > Martin Kou
> >
> >
> > On Thu, Aug 8, 2013 at 10:37 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > We have a 3-node ZK cluster running 3.3.4 that are experiencing high
> > > latency. From "stat", we saw avg latency to be 300-400ms consistently.
> > > Outstanding requests jumps to ~600 from time to time. However, the
> > server's
> > > I/O and CPU are both low. If I start a new ZK client and do a few
> writes,
> > > the observed response time in the client is also low (1ms).
> > >
> > > What's the best way to figure out why the reported latency in ZK server
> > is
> > > high? Also, are  min/avg/max latency computed on a rolling time window?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB