Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka on EC2


Copy link to this message
-
Re: Kafka on EC2
We use m1.large's with ephemeral storage and get 20MB/sec using Kafka's
built in benchmarking tool.   No compression.

On Tue, Nov 20, 2012 at 7:52 AM, David Arthur <[EMAIL PROTECTED]> wrote:

> In my experience, anything smaller than m1.xlarge isn't really suitable
> for I/O intensive high performance stuff. I would guess that, for Kafka, a
> single m1.xlarge would outperform two m1.large. I have no hard evidence to
> support this however.
>
> What I'd like to see are some benchmarks comparing 12 m1.large to 6
> m1.xlarge to 1 hi1.4xlarge.
>
> Another interesting note is with the m1.xlarge you can get "optimized" EBS
> instances with a claimed 1000 Mbps I/O throughput.
>
> On Nov 19, 2012, at 7:36 PM, Bae, Jae Hyeon wrote:
>
> > Yes, 12 m1.large instances couldn't handle more than 12k messages per
> > second in our environment. When the traffic goes up to 12k/sec, kafka
> > clusters started to be throttling. I am not sure how much one
> > m1.xlarge will outperform 2 m1.large instances because m1.xlarge is
> > twice expensive than m1.large. I vote that two m1.large will be better
> > than 1 m1.xlarge.
> >
> > Speaking of EBS volume, EBS writing performance is not good and it's
> > expensive. I hope replication in 0.8 will save us.
> >
> > On Sun, Nov 18, 2012 at 3:36 PM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
> >>>> Single m1.large instance's capacity was 10k/sec.
> >>
> >> When you say capacity, did you mean the I/O or network capacity on the
> >> m1.large instances ?
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Sun, Nov 18, 2012 at 3:26 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]>
> wrote:
> >>> I am running kafka on ec2 with m1.large instance. I think that large
> >>> number of low end servers will outperform small number of high end
> >>> servers, but I am not sure I am correct.
> >>>
> >>> I assumed 12 m1.large instances would be able to handle more than 6
> >>> billion rows in a day but my expectation was wrong. Single m1.large
> >>> instance's capacity was 10k/sec. If we want to have a room, we'd
> >>> better assume kafka on m1.large maximum capacity is 8k/sec.
> >>>
> >>> On Fri, Nov 16, 2012 at 5:57 PM, Joel Koshy <[EMAIL PROTECTED]>
> wrote:
> >>>> At least based on prior threads (discussing experiences/issues with
> ec2),
> >>>> there should be a number of people on this list who can help you. It
> would
> >>>> be helpful if we have a ec2-operations wiki @
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/Index . Would
> people be
> >>>> interested in sharing operational experiences there? I'm thinking
> something
> >>>> similar to the operations wiki that's already available (
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/Operations).
> >>>>
> >>>> Joel
> >>>>
> >>>>
> >>>> On Fri, Nov 16, 2012 at 5:04 PM, Senthilvel Rangaswamy <
> [EMAIL PROTECTED]
> >>>>> wrote:
> >>>>
> >>>>> Have folks implemented large installations on Kafka on Amazon EC2. I
> am
> >>>>> looking for best practices. Like the kind of nodes, EBS vs Instance
> store
> >>>>> etc.,
> >>>>>
> >>>>> --
> >>>>> ..Senthil
> >>>>>
> >>>>> "If there's anything more important than my ego around, I want it
> >>>>> caught and shot now."
> >>>>>                                                    - Douglas Adams.
> >>>>>
>
>
--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>