Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - ec2 suggestion setup for a minimum kafka setup; zookeeper is bad in ec2?


Copy link to this message
-
ec2 suggestion setup for a minimum kafka setup; zookeeper is bad in ec2?
S Ahmed 2014-06-11, 17:43
For those of you hosting on ec2, could someone suggest a "minimum"
recommended setup for kafka?  i.e. the # and type of instance size that you
would say is the bare minimum to get started with kafka in ec2.

My guess is the suggest route is the m3 instance type?
How about:

m3.medium  1 cpu, 3.75GB Ram  4 GB SSD
m3.large       2 cpus, 7.5 GB ram   32 GB SSD
My requirements are maybe a few hundred messages per second to start, but
want to grow my usage as my confidence/experience with kafka increases.

I was reading this post regarding kafka on ec2 (
http://engineering.onlive.com/2013/12/12/didnt-use-kafka/) and one of the
main cons the author says exists with kafka on ec2 is that zookeeper is a
little finicky when it comes to latency and may knock a host out b/c of
latency issues.

How do you guys on ec2 work around these issues?