Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Loggly's use of Kafka on AWS


+
Philip OToole 2013-12-02, 02:17
+
Joe Stein 2013-12-02, 02:59
+
S Ahmed 2013-12-02, 22:50
+
Surendranauth Hiraman 2013-12-02, 22:56
+
Steve Morin 2013-12-03, 01:16
+
Timothy Chen 2013-12-03, 15:38
+
Philip OToole 2013-12-03, 15:41
Copy link to this message
-
Re: Loggly's use of Kafka on AWS
Jonathan Hodges 2013-12-05, 13:40
A cheaper approach is to use ephemeral disk and have the replicas across
AZs.  You can lose up to 2 AZs and still avoid data loss.  You mitigate
complete region failure using mirror maker to replicate to another region.
On Tue, Dec 3, 2013 at 8:41 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:

> Yeah, we use provisioned IOPs EBS so we have good performance.
>
> We run 3 ZK nodes, one in each AZ. Quorum 2.
>
> Philip
>
> On Dec 3, 2013, at 7:31 AM, Timothy Chen <[EMAIL PROTECTED]> wrote:
>
> > Hi Philip,
> >
> > So I wonder if you guys hit disk perf problems with EBS? It seems quite
> common in the past but I haven't tried recently.
> >
> > Also can you share how you guys deployed zookeeper in AWS so that a
> qurom is always available?
> >
> > Tim
> >
> > Sent from my iPhone
> >
> >> On Dec 2, 2013, at 5:15 PM, Steve Morin <[EMAIL PROTECTED]> wrote:
> >>
> >> Philip this is definitely useful
> >>
> >>> On Dec 2, 2013, at 14:55, Surendranauth Hiraman <
> [EMAIL PROTECTED]> wrote:
> >>>
> >>> S Ahmed,
> >>>
> >>> This combination of Kafka and Storm to process streaming data is
> becoming
> >>> pretty common. Definitely worth looking at.
> >>>
> >>> The throughput will vary depending on your workload (cpu usage, etc.)
> and
> >>> if you're talking to a backend, of course. But it scales very well.
> >>>
> >>> -Suren
> >>>
> >>>
> >>>
> >>>> On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>> Interesting.  So twitter storm is used to basically process the
> messages on
> >>>> kafka?   I'll have to read-up on storm b/c I always thought the use
> case
> >>>> was a bit different.
> >>>>
> >>>>
> >>>>
> >>>>> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <[EMAIL PROTECTED]>
> wrote:
> >>>>>
> >>>>> Awesome Philip, thanks for sharing!
> >>>>>
> >>>>>> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <[EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>
> >>>>>> A couple of us here at Loggly recently spoke at AWS reinvent, on
> how we
> >>>>>> use Kafka 0.72 in our ingestion pipeline. The slides are at the link
> >>>>> below,
> >>>>>> and may be of interest to people on this list.
> >>>>
> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
> >>>>>>
> >>>>>> Any questions, let me know, though I can't promise I can answer
> >>>>>> everything. Can't give the complete game away. :-)
> >>>>>>
> >>>>>> As always, Kafka rocks!
> >>>>>>
> >>>>>> Philip
> >>>
> >>>
> >>>
> >>> --
> >>> ___________________________________________
> >>> Available at these partners:
> >>> [image: CloudFlare | shopify | Bigcommerce]
> >>>
> >>> SUREN HIRAMAN, VP TECHNOLOGY
> >>> SOCIOCAST
> >>> Simple. Powerful. Predictions.
> >>>
> >>> 96 SPRING STREET, 7TH FLOOR
> >>> NEW YORK, NY 10012
> >>> O: (917) 525-2466 ext. 105
> >>> F: 646.349.4063
> >>> E: [EMAIL PROTECTED]
> >>> W: www.sociocast.com
> >>>
> >>> Increase Conversion Rates up to 500%. Go to www.sociocast.com and
> enter
> >>> your URL for a free trial!
>