Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Loggly's use of Kafka on AWS


Copy link to this message
-
Re: Loggly's use of Kafka on AWS
Hi Philip,

So I wonder if you guys hit disk perf problems with EBS? It seems quite common in the past but I haven't tried recently.

Also can you share how you guys deployed zookeeper in AWS so that a qurom is always available?

Tim

Sent from my iPhone

> On Dec 2, 2013, at 5:15 PM, Steve Morin <[EMAIL PROTECTED]> wrote:
>
> Philip this is definitely useful
>
>> On Dec 2, 2013, at 14:55, Surendranauth Hiraman <[EMAIL PROTECTED]> wrote:
>>
>> S Ahmed,
>>
>> This combination of Kafka and Storm to process streaming data is becoming
>> pretty common. Definitely worth looking at.
>>
>> The throughput will vary depending on your workload (cpu usage, etc.) and
>> if you're talking to a backend, of course. But it scales very well.
>>
>> -Suren
>>
>>
>>
>>> On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
>>>
>>> Interesting.  So twitter storm is used to basically process the messages on
>>> kafka?   I'll have to read-up on storm b/c I always thought the use case
>>> was a bit different.
>>>
>>>
>>>
>>>> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Awesome Philip, thanks for sharing!
>>>>
>>>>> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>>> A couple of us here at Loggly recently spoke at AWS reinvent, on how we
>>>>> use Kafka 0.72 in our ingestion pipeline. The slides are at the link
>>>> below,
>>>>> and may be of interest to people on this list.
>>> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
>>>>>
>>>>> Any questions, let me know, though I can't promise I can answer
>>>>> everything. Can't give the complete game away. :-)
>>>>>
>>>>> As always, Kafka rocks!
>>>>>
>>>>> Philip
>>
>>
>>
>> --
>> ___________________________________________
>> Available at these partners:
>> [image: CloudFlare | shopify | Bigcommerce]
>>
>> SUREN HIRAMAN, VP TECHNOLOGY
>> SOCIOCAST
>> Simple. Powerful. Predictions.
>>
>> 96 SPRING STREET, 7TH FLOOR
>> NEW YORK, NY 10012
>> O: (917) 525-2466 ext. 105
>> F: 646.349.4063
>> E: [EMAIL PROTECTED]
>> W: www.sociocast.com
>>
>> Increase Conversion Rates up to 500%. Go to www.sociocast.com and enter
>> your URL for a free trial!