In setting up an Amazon EC2 configuration for a group of zookeeper nodes
I'm wondering about disk configurations.  The zk documentation stresses the
advantages of a dedicated disk for the transaction log.  Does this imply
that two smaller disks would be more performant than a single larger disk?
 I'm anticipating a 5 node write-heavy cluster.

Thanks.

--
http://about.me/BrianTarbox

Two disks could be a significant advantage.

You should also experiment with ways to avoid VM induced gaps in time.

Finally, if you really are going to be write heavy, 3 nodes are likely to
perform better than 5.

On Tue, Oct 9, 2012 at 2:03 PM, Brian Tarbox <[EMAIL PROTECTED]> wrote:

> In setting up an Amazon EC2 configuration for a group of zookeeper nodes
> I'm wondering about disk configurations.  The zk documentation stresses the
> advantages of a dedicated disk for the transaction log.  Does this imply
> that two smaller disks would be more performant than a single larger disk?
>  I'm anticipating a 5 node write-heavy cluster.
>
> Thanks.
>
> --
> http://about.me/BrianTarbox
>

I'd also add that EC2 supports a variety of different ways of configuring the disks which should be explored if you are very write heavy. They include:
* Using high IOPS instances
* Using multiple disks and doing OS level RAID across them to present a single logical disk to the app
* Local vs. EBS volumes
* Standard vs. provisioned IOPS w/EBS

I'd be interested to know if others have already done benchmarking of these different configurations. I've never heard about it if so (but I’m pretty new to ZK so that doesn't mean much :)).

~Eric

-----Original Message-----
From: Ted Dunning [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, October 09, 2012 8:29 AM
To: [EMAIL PROTECTED]
Subject: Re: EC2 disk configuration for zookeeper? One big disk or two smaller ones per node?

Two disks could be a significant advantage.

You should also experiment with ways to avoid VM induced gaps in time.

Finally, if you really are going to be write heavy, 3 nodes are likely to perform better than 5.

On Tue, Oct 9, 2012 at 2:03 PM, Brian Tarbox <[EMAIL PROTECTED]> wrote:

> In setting up an Amazon EC2 configuration for a group of zookeeper
> nodes I'm wondering about disk configurations.  The zk documentation
> stresses the advantages of a dedicated disk for the transaction log.  
> Does this imply that two smaller disks would be more performant than a single larger disk?
>  I'm anticipating a 5 node write-heavy cluster.
>
> Thanks.
>
> --
> http://about.me/BrianTarbox
>

I have some experience with these.

On Tue, Oct 9, 2012 at 4:47 PM, Eric Fleischman <[EMAIL PROTECTED]>wrote:

> * Using high IOPS instances
>

Haven't tried this.
> * Using multiple disks and doing OS level RAID across them to present a
> single logical disk to the app
>

This typically is very dangerous unless you can somehow arrange for disks
not to present lowest common denominator performance.
> * Local vs. EBS volumes
>

Local tends to be much lower variance.  EBS can show 10:1 variations in
performance.  Even the same volume can change dramatically over time.

 * Standard vs. provisioned IOPS w/EBS
>

Haven't tried this.

I can ack Ted's experience.

Haven't tried the provisioned IOPS EBS volumes but if you want to try
those, be sure to select the "reserved IOPS optimized" instances or else
your storage traffic will still be multiplexed with instance networking.

On Wednesday, October 10, 2012, Ted Dunning wrote:

> I have some experience with these.
>
> On Tue, Oct 9, 2012 at 4:47 PM, Eric Fleischman <[EMAIL PROTECTED]<javascript:;>
> >wrote:
>
> > * Using high IOPS instances
> >
>
> Haven't tried this.
>
>
> > * Using multiple disks and doing OS level RAID across them to present a
> > single logical disk to the app
> >
>
> This typically is very dangerous unless you can somehow arrange for disks
> not to present lowest common denominator performance.
>
>
> > * Local vs. EBS volumes
> >
>
> Local tends to be much lower variance.  EBS can show 10:1 variations in
> performance.  Even the same volume can change dramatically over time.
>
>  * Standard vs. provisioned IOPS w/EBS
> >
>
> Haven't tried this.
>
--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

> > * Using multiple disks and doing OS level RAID across them to present a
> > single logical disk to the app
> >
>
> This typically is very dangerous unless you can somehow arrange for disks
> not to present lowest common denominator performance.
>

Not to mention the lower reliability implications of the setup.

We've tried this setup with local disks on EC2 with decent performance results.  We have done this only in the context of testing a write heavy application (not directly ZK) with medium to large size workloads, but I would not recommend this setup for production unless there's another form of replication.  It may be OK in the context of a 3 or more node ZK cluster.

Also, when using local disks on EC2, beware of the "first write problem".  The first time a range is written to the device is slow, subsequent writes can easily be 2X to 3X faster.  To 'get around this', we first write to the device (this takes a long time), before using it for our tests.

I hope this helps,

-Julio