Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> EC2 disk configuration for zookeeper? One big disk or two smaller ones per node?


+
Brian Tarbox 2012-10-09, 13:03
+
Ted Dunning 2012-10-09, 15:29
+
Eric Fleischman 2012-10-09, 15:47
+
Ted Dunning 2012-10-09, 16:44
+
Andrew Purtell 2012-10-10, 00:06
Copy link to this message
-
Re: Re: EC2 disk configuration for zookeeper? One big disk or two smaller ones per node?


> > * Using multiple disks and doing OS level RAID across them to present a
> > single logical disk to the app
> >
>
> This typically is very dangerous unless you can somehow arrange for disks
> not to present lowest common denominator performance.
>

Not to mention the lower reliability implications of the setup.

We've tried this setup with local disks on EC2 with decent performance results.  We have done this only in the context of testing a write heavy application (not directly ZK) with medium to large size workloads, but I would not recommend this setup for production unless there's another form of replication.  It may be OK in the context of a 3 or more node ZK cluster.

Also, when using local disks on EC2, beware of the "first write problem".  The first time a range is written to the device is slow, subsequent writes can easily be 2X to 3X faster.  To 'get around this', we first write to the device (this takes a long time), before using it for our tests.

I hope this helps,

-Julio