Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka on EC2


Copy link to this message
-
Re: Kafka on EC2
I'd only consider m1.xlarge and higher for Kafka. The m1.xlarge have "high" I/O performance according to Amazon. This is disk I/O and network I/O performance. Of course you need to use EBS volumes if you want your Kafka brokers to survive reboots - you can expect reboots on AWS. Some people have reported I/O improvements by RAIDing EBS volumes (http://alestic.com/2009/06/ec2-ebs-raid). Deploying in the same region as your application will also improve performance.
On Nov 16, 2012, at 8:04 PM, Senthilvel Rangaswamy wrote:

> Have folks implemented large installations on Kafka on Amazon EC2. I am
> looking for best practices. Like the kind of nodes, EBS vs Instance store
> etc.,
>
> --
> ..Senthil
>
> "If there's anything more important than my ego around, I want it
> caught and shot now."
>                                                    - Douglas Adams.

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB