Senthilvel Rangaswamy 2012-11-17, 01:04
Joel Koshy 2012-11-17, 01:57
Bae, Jae Hyeon 2012-11-18, 23:26
Neha Narkhede 2012-11-18, 23:36
Bae, Jae Hyeon 2012-11-20, 00:36
-Re: Kafka on EC2
David Arthur 2012-11-20, 15:52
In my experience, anything smaller than m1.xlarge isn't really suitable for I/O intensive high performance stuff. I would guess that, for Kafka, a single m1.xlarge would outperform two m1.large. I have no hard evidence to support this however.
What I'd like to see are some benchmarks comparing 12 m1.large to 6 m1.xlarge to 1 hi1.4xlarge.
Another interesting note is with the m1.xlarge you can get "optimized" EBS instances with a claimed 1000 Mbps I/O throughput.
On Nov 19, 2012, at 7:36 PM, Bae, Jae Hyeon wrote:
> Yes, 12 m1.large instances couldn't handle more than 12k messages per
> second in our environment. When the traffic goes up to 12k/sec, kafka
> clusters started to be throttling. I am not sure how much one
> m1.xlarge will outperform 2 m1.large instances because m1.xlarge is
> twice expensive than m1.large. I vote that two m1.large will be better
> than 1 m1.xlarge.
> Speaking of EBS volume, EBS writing performance is not good and it's
> expensive. I hope replication in 0.8 will save us.
> On Sun, Nov 18, 2012 at 3:36 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:
>>>> Single m1.large instance's capacity was 10k/sec.
>> When you say capacity, did you mean the I/O or network capacity on the
>> m1.large instances ?
>> On Sun, Nov 18, 2012 at 3:26 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]> wrote:
>>> I am running kafka on ec2 with m1.large instance. I think that large
>>> number of low end servers will outperform small number of high end
>>> servers, but I am not sure I am correct.
>>> I assumed 12 m1.large instances would be able to handle more than 6
>>> billion rows in a day but my expectation was wrong. Single m1.large
>>> instance's capacity was 10k/sec. If we want to have a room, we'd
>>> better assume kafka on m1.large maximum capacity is 8k/sec.
>>> On Fri, Nov 16, 2012 at 5:57 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>>>> At least based on prior threads (discussing experiences/issues with ec2),
>>>> there should be a number of people on this list who can help you. It would
>>>> be helpful if we have a ec2-operations wiki @
>>>> https://cwiki.apache.org/confluence/display/KAFKA/Index . Would people be
>>>> interested in sharing operational experiences there? I'm thinking something
>>>> similar to the operations wiki that's already available (
>>>> On Fri, Nov 16, 2012 at 5:04 PM, Senthilvel Rangaswamy <[EMAIL PROTECTED]
>>>>> Have folks implemented large installations on Kafka on Amazon EC2. I am
>>>>> looking for best practices. Like the kind of nodes, EBS vs Instance store
>>>>> "If there's anything more important than my ego around, I want it
>>>>> caught and shot now."
>>>>> - Douglas Adams.
Evan Chan 2012-11-20, 17:12
David Arthur 2012-11-19, 19:30
Matthew Rathbone 2012-11-19, 19:44
James A. Robinson 2012-11-19, 19:56
Matt Jones 2012-11-19, 21:34
Matthew Rathbone 2012-11-19, 21:44
Senthilvel Rangaswamy 2012-11-19, 20:17