Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> RE: [jira] [Commented] (ZOOKEEPER-866) Adding no disk persistence option in zookeeper.

Copy link to this message
RE: [jira] [Commented] (ZOOKEEPER-866) Adding no disk persistence option in zookeeper.
hi Graham,

Could you please update with some more details like the size of znode created and some of the system parameters like size of the ramdisk, size of RAM etc.. I guess this would give a better idea to compare your results with ours and discuss on improvements.

Sent: Wednesday, November 21, 2012 2:57 PM
Subject: [jira] [Commented] (ZOOKEEPER-866) Adding no disk persistence option in zookeeper.

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502200#comment-13502200 ]

Graham commented on ZOOKEEPER-866:

We ran some benchmarks using zk-latencies.

We tried to ways to boost performance: Using a RAM disk (tmpfs) and using libeatmydata (Makes all file system sync operations no-ops).

libeatmydata benchmarks: http://pastebin.com/cNLjfhPG

Ramdisk using tmpfs benchmarks: http://pastebin.com/mfe92nXn (Note: Different box to the last one)

In summary: Synchronous calls are boosted by two orders of magnitude with either libeatmydata or ramdisk (In Standalone mode and also clustered mode). Asynchronous calls are boosted by a factor of 2 or 3.

For tests, simulations etc. a Zookeeper without snapshots or logs makes a lot of sense, but for production use, the ramdisk or eatmydata options both looks pretty good.

Another thing we found works well is to have a battery backed raid array; writes to go to raid cache and will sync to disk eventually.

> Adding no disk persistence option in zookeeper.
> -----------------------------------------------
>                 Key: ZOOKEEPER-866
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-866
>             Project: ZooKeeper
>          Issue Type: New Feature
>            Reporter: Mahadev konar
>            Assignee: Mahadev konar
>             Fix For: 3.5.0
>         Attachments: ZOOKEEPER-nodisk.patch
> Its been seen that some folks would like to use zookeeper for very fine grained locking. Also, in there use case they are fine with loosing all old zookeeper state if they reboot zookeeper or zookeeper goes down. The use case is more of a runtime locking wherein forgetting the state of locks is acceptable in case of a zookeeper reboot. Not logging to disk allows high throughput on and low latency on the writes to zookeeper. This would be a configuration option to set (ofcourse the default would be logging to disk).

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira