Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka on EC2


Copy link to this message
-
Re: Kafka on EC2
We use m1.large's with ephemeral storage and get 20MB/sec using Kafka's
built in benchmarking tool.   No compression.

On Tue, Nov 20, 2012 at 7:52 AM, David Arthur <[EMAIL PROTECTED]> wrote:

> In my experience, anything smaller than m1.xlarge isn't really suitable
> for I/O intensive high performance stuff. I would guess that, for Kafka, a
> single m1.xlarge would outperform two m1.large. I have no hard evidence to
> support this however.
>
> What I'd like to see are some benchmarks comparing 12 m1.large to 6
> m1.xlarge to 1 hi1.4xlarge.
>
> Another interesting note is with the m1.xlarge you can get "optimized" EBS
> instances with a claimed 1000 Mbps I/O throughput.
>
> On Nov 19, 2012, at 7:36 PM, Bae, Jae Hyeon wrote:
>
> > Yes, 12 m1.large instances couldn't handle more than 12k messages per
> > second in our environment. When the traffic goes up to 12k/sec, kafka
> > clusters started to be throttling. I am not sure how much one
> > m1.xlarge will outperform 2 m1.large instances because m1.xlarge is
> > twice expensive than m1.large. I vote that two m1.large will be better
> > than 1 m1.xlarge.
> >
> > Speaking of EBS volume, EBS writing performance is not good and it's
> > expensive. I hope replication in 0.8 will save us.
> >
> > On Sun, Nov 18, 2012 at 3:36 PM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
> >>>> Single m1.large instance's capacity was 10k/sec.
> >>
> >> When you say capacity, did you mean the I/O or network capacity on the
> >> m1.large instances ?
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Sun, Nov 18, 2012 at 3:26 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]>
> wrote:
> >>> I am running kafka on ec2 with m1.large instance. I think that large
> >>> number of low end servers will outperform small number of high end
> >>> servers, but I am not sure I am correct.
> >>>
> >>> I assumed 12 m1.large instances would be able to handle more than 6
> >>> billion rows in a day but my expectation was wrong. Single m1.large
> >>> instance's capacity was 10k/sec. If we want to have a room, we'd
> >>> better assume kafka on m1.large maximum capacity is 8k/sec.
> >>>
> >>> On Fri, Nov 16, 2012 at 5:57 PM, Joel Koshy <[EMAIL PROTECTED]>
> wrote:
> >>>> At least based on prior threads (discussing experiences/issues with
> ec2),
> >>>> there should be a number of people on this list who can help you. It
> would
> >>>> be helpful if we have a ec2-operations wiki @
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/Index . Would
> people be
> >>>> interested in sharing operational experiences there? I'm thinking
> something
> >>>> similar to the operations wiki that's already available (
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/Operations).
> >>>>
> >>>> Joel
> >>>>
> >>>>
> >>>> On Fri, Nov 16, 2012 at 5:04 PM, Senthilvel Rangaswamy <
> [EMAIL PROTECTED]
> >>>>> wrote:
> >>>>
> >>>>> Have folks implemented large installations on Kafka on Amazon EC2. I
> am
> >>>>> looking for best practices. Like the kind of nodes, EBS vs Instance
> store
> >>>>> etc.,
> >>>>>
> >>>>> --
> >>>>> ..Senthil
> >>>>>
> >>>>> "If there's anything more important than my ego around, I want it
> >>>>> caught and shot now."
> >>>>>                                                    - Douglas Adams.
> >>>>>
>
>
--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB