Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Recovery log on /dev/shm?


Copy link to this message
-
Re: Recovery log on /dev/shm?
I had done quite a bit of benchmarks on not logging to disk.
See ZOOKEEPER-866

and noticed that it actually wasnt of much help. It doesnt improve the latency by much and also the throughput was still the same.

hope that helps.

mahadev

On Sep 1, 2011, at 2:24 PM, Benjamin Reed wrote:

> have you benchmarked the difference between using the ram disk and
> turning off sync with forceSync=no? i would think that the performance
> will be similar. you will need to use the code in trunk because
> https://issues.apache.org/jira/browse/ZOOKEEPER-1087 isn't in any of
> the releases yet.
>
> thanx
> ben
>
> On Thu, Sep 1, 2011 at 1:31 PM, Ashwin Jayaprakash
> <[EMAIL PROTECTED]> wrote:
>> This is a follow up to a question I posted in July 2010 - about whether the
>> recovery log could be redirected to /dev/null or some such low latency drive
>> [http://markmail.org/thread/3edpfmvn2fhyg5wv].
>>
>> I just found out that Linux distros with 2.6 kernel have a built-in shared
>> memory file system that runs purely in memory - /dev/shm. Since this
>> provides recovery across JVM restarts (but not OS restarts) wouldn't this be
>> sufficient to run the ZK nodes off this drive? Replicas are on other
>> machines anyway so FT/HA is already there. It's just that disk latency can
>> be eliminated completely.
>>
>> [Apologies if you guys are already aware of /dev/shm]
>>
>> I ran some simple Java IO tests (not full ZK) on this drive and I wanted to
>> share my results. Might be of some use to apps that require low latency:
>> http://javaforu.blogspot.com/2011/09/ram-disk-is-already-in-linux-and-nobody.html
>>
>> Ashwin.
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB