Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> OperationTimeoutException error


+
Deepinder Singh Setia 2013-08-09, 16:13
Copy link to this message
-
Re: OperationTimeoutException error
Hi.

On Fri, Aug 9, 2013, at 18:13, Deepinder Singh Setia wrote:
> Aug  9 07:07:20 a2s1 python[2085]: OperationTimeoutException: operation
> timeout

That's one of the "retryable exceptions" in Kazoo. So if you'd use
client.retry, you could tolerate one or more instances of this error.

> zookeeper logs around the error time:
>
> 2013-08-09 07:07:06,580 [myid:] - WARN  [SyncThread:0:FileTxnLog@321] -
> fsync-ing the write ahead log in SyncThread:0 took 2291ms which will
> adversely effect operation latency. See the ZooKeeper troubleshooting
> guide

More than 2 seconds of fsync stall is quite long. And with that or GC
pauses, it's more than likely that you exceed the session timeout
limits.

Did you follow the recommendations in
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html? Especially
around using dedicated disks for the transaction log and using a
dedicated machine for Zookeeper to avoid other processes stalling it?

> Could the client (Kazoo) be timing out because of fsync delay? What
> parameter would control duration for OperationTimeoutException that I can
> perhaps increase to verify? There is only ZooKeeper client and the load
> isn't much - 1 read/sec and 2 writes/sec roughly. Zookeeper configuration
> is default. Kazoo client params are also default.

In the admin guide, look at tickTime and syncLimit. In a default config
the session timeout is ~4 seconds. While you can increase this value,
you thereby also increase the minimum time it takes Zookeeper to
consider an actual client to be dead. Depending on what you use ZK for,
you might prefer failing fast and thus low session timeout values.

Hanno
+
Deepinder Singh Setia 2013-08-14, 17:11
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB