Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka on EC2

Copy link to this message
Re: Kafka on EC2
We RAID-0 4 EBS disks, we find that to be most performant, although it does
leave you more vulnerable to EBS network errors and outages. Ideally, if
you're spread across AZ's you could do some clever routing to
geographically local brokers, whilst keeping the others as a backup in case
of failure.

In the same vein, we have N+1 brokers, where N is how many we reasonably
think we need, this way we can hopefully survive outages.
On Mon, Nov 19, 2012 at 1:30 PM, David Arthur <[EMAIL PROTECTED]> wrote:

> I'd only consider m1.xlarge and higher for Kafka. The m1.xlarge have
> "high" I/O performance according to Amazon. This is disk I/O and network
> I/O performance. Of course you need to use EBS volumes if you want your
> Kafka brokers to survive reboots - you can expect reboots on AWS. Some
> people have reported I/O improvements by RAIDing EBS volumes (
> http://alestic.com/2009/06/ec2-ebs-raid). Deploying in the same region as
> your application will also improve performance.
> On Nov 16, 2012, at 8:04 PM, Senthilvel Rangaswamy wrote:
> > Have folks implemented large installations on Kafka on Amazon EC2. I am
> > looking for best practices. Like the kind of nodes, EBS vs Instance store
> > etc.,
> >
> > --
> > ..Senthil
> >
> > "If there's anything more important than my ego around, I want it
> > caught and shot now."
> >                                                    - Douglas Adams.
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |