Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka on EC2


Copy link to this message
-
Re: Kafka on EC2
David Arthur 2012-11-20, 15:52
In my experience, anything smaller than m1.xlarge isn't really suitable for I/O intensive high performance stuff. I would guess that, for Kafka, a single m1.xlarge would outperform two m1.large. I have no hard evidence to support this however.

What I'd like to see are some benchmarks comparing 12 m1.large to 6 m1.xlarge to 1 hi1.4xlarge.

Another interesting note is with the m1.xlarge you can get "optimized" EBS instances with a claimed 1000 Mbps I/O throughput.

On Nov 19, 2012, at 7:36 PM, Bae, Jae Hyeon wrote:

> Yes, 12 m1.large instances couldn't handle more than 12k messages per
> second in our environment. When the traffic goes up to 12k/sec, kafka
> clusters started to be throttling. I am not sure how much one
> m1.xlarge will outperform 2 m1.large instances because m1.xlarge is
> twice expensive than m1.large. I vote that two m1.large will be better
> than 1 m1.xlarge.
>
> Speaking of EBS volume, EBS writing performance is not good and it's
> expensive. I hope replication in 0.8 will save us.
>
> On Sun, Nov 18, 2012 at 3:36 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:
>>>> Single m1.large instance's capacity was 10k/sec.
>>
>> When you say capacity, did you mean the I/O or network capacity on the
>> m1.large instances ?
>>
>> Thanks,
>> Neha
>>
>>
>> On Sun, Nov 18, 2012 at 3:26 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]> wrote:
>>> I am running kafka on ec2 with m1.large instance. I think that large
>>> number of low end servers will outperform small number of high end
>>> servers, but I am not sure I am correct.
>>>
>>> I assumed 12 m1.large instances would be able to handle more than 6
>>> billion rows in a day but my expectation was wrong. Single m1.large
>>> instance's capacity was 10k/sec. If we want to have a room, we'd
>>> better assume kafka on m1.large maximum capacity is 8k/sec.
>>>
>>> On Fri, Nov 16, 2012 at 5:57 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>>>> At least based on prior threads (discussing experiences/issues with ec2),
>>>> there should be a number of people on this list who can help you. It would
>>>> be helpful if we have a ec2-operations wiki @
>>>> https://cwiki.apache.org/confluence/display/KAFKA/Index . Would people be
>>>> interested in sharing operational experiences there? I'm thinking something
>>>> similar to the operations wiki that's already available (
>>>> https://cwiki.apache.org/confluence/display/KAFKA/Operations).
>>>>
>>>> Joel
>>>>
>>>>
>>>> On Fri, Nov 16, 2012 at 5:04 PM, Senthilvel Rangaswamy <[EMAIL PROTECTED]
>>>>> wrote:
>>>>
>>>>> Have folks implemented large installations on Kafka on Amazon EC2. I am
>>>>> looking for best practices. Like the kind of nodes, EBS vs Instance store
>>>>> etc.,
>>>>>
>>>>> --
>>>>> ..Senthil
>>>>>
>>>>> "If there's anything more important than my ego around, I want it
>>>>> caught and shot now."
>>>>>                                                    - Douglas Adams.
>>>>>