Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Loggly's use of Kafka on AWS


Copy link to this message
-
Re: Loggly's use of Kafka on AWS
Yeah, we use provisioned IOPs EBS so we have good performance.

We run 3 ZK nodes, one in each AZ. Quorum 2.

Philip

On Dec 3, 2013, at 7:31 AM, Timothy Chen <[EMAIL PROTECTED]> wrote:

> Hi Philip,
>
> So I wonder if you guys hit disk perf problems with EBS? It seems quite common in the past but I haven't tried recently.
>
> Also can you share how you guys deployed zookeeper in AWS so that a qurom is always available?
>
> Tim
>
> Sent from my iPhone
>
>> On Dec 2, 2013, at 5:15 PM, Steve Morin <[EMAIL PROTECTED]> wrote:
>>
>> Philip this is definitely useful
>>
>>> On Dec 2, 2013, at 14:55, Surendranauth Hiraman <[EMAIL PROTECTED]> wrote:
>>>
>>> S Ahmed,
>>>
>>> This combination of Kafka and Storm to process streaming data is becoming
>>> pretty common. Definitely worth looking at.
>>>
>>> The throughput will vary depending on your workload (cpu usage, etc.) and
>>> if you're talking to a backend, of course. But it scales very well.
>>>
>>> -Suren
>>>
>>>
>>>
>>>> On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Interesting.  So twitter storm is used to basically process the messages on
>>>> kafka?   I'll have to read-up on storm b/c I always thought the use case
>>>> was a bit different.
>>>>
>>>>
>>>>
>>>>> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>> Awesome Philip, thanks for sharing!
>>>>>
>>>>>> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>
>>>>>> A couple of us here at Loggly recently spoke at AWS reinvent, on how we
>>>>>> use Kafka 0.72 in our ingestion pipeline. The slides are at the link
>>>>> below,
>>>>>> and may be of interest to people on this list.
>>>> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
>>>>>>
>>>>>> Any questions, let me know, though I can't promise I can answer
>>>>>> everything. Can't give the complete game away. :-)
>>>>>>
>>>>>> As always, Kafka rocks!
>>>>>>
>>>>>> Philip
>>>
>>>
>>>
>>> --
>>> ___________________________________________
>>> Available at these partners:
>>> [image: CloudFlare | shopify | Bigcommerce]
>>>
>>> SUREN HIRAMAN, VP TECHNOLOGY
>>> SOCIOCAST
>>> Simple. Powerful. Predictions.
>>>
>>> 96 SPRING STREET, 7TH FLOOR
>>> NEW YORK, NY 10012
>>> O: (917) 525-2466 ext. 105
>>> F: 646.349.4063
>>> E: [EMAIL PROTECTED]
>>> W: www.sociocast.com
>>>
>>> Increase Conversion Rates up to 500%. Go to www.sociocast.com and enter
>>> your URL for a free trial!