Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Loggly's use of Kafka on AWS


+
Philip OToole 2013-12-02, 02:17
+
Joe Stein 2013-12-02, 02:59
+
S Ahmed 2013-12-02, 22:50
+
Surendranauth Hiraman 2013-12-02, 22:56
+
Steve Morin 2013-12-03, 01:16
+
Timothy Chen 2013-12-03, 15:38
+
Philip OToole 2013-12-03, 15:41
Copy link to this message
-
Re: Loggly's use of Kafka on AWS
A cheaper approach is to use ephemeral disk and have the replicas across
AZs.  You can lose up to 2 AZs and still avoid data loss.  You mitigate
complete region failure using mirror maker to replicate to another region.
On Tue, Dec 3, 2013 at 8:41 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:

> Yeah, we use provisioned IOPs EBS so we have good performance.
>
> We run 3 ZK nodes, one in each AZ. Quorum 2.
>
> Philip
>
> On Dec 3, 2013, at 7:31 AM, Timothy Chen <[EMAIL PROTECTED]> wrote:
>
> > Hi Philip,
> >
> > So I wonder if you guys hit disk perf problems with EBS? It seems quite
> common in the past but I haven't tried recently.
> >
> > Also can you share how you guys deployed zookeeper in AWS so that a
> qurom is always available?
> >
> > Tim
> >
> > Sent from my iPhone
> >
> >> On Dec 2, 2013, at 5:15 PM, Steve Morin <[EMAIL PROTECTED]> wrote:
> >>
> >> Philip this is definitely useful
> >>
> >>> On Dec 2, 2013, at 14:55, Surendranauth Hiraman <
> [EMAIL PROTECTED]> wrote:
> >>>
> >>> S Ahmed,
> >>>
> >>> This combination of Kafka and Storm to process streaming data is
> becoming
> >>> pretty common. Definitely worth looking at.
> >>>
> >>> The throughput will vary depending on your workload (cpu usage, etc.)
> and
> >>> if you're talking to a backend, of course. But it scales very well.
> >>>
> >>> -Suren
> >>>
> >>>
> >>>
> >>>> On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>> Interesting.  So twitter storm is used to basically process the
> messages on
> >>>> kafka?   I'll have to read-up on storm b/c I always thought the use
> case
> >>>> was a bit different.
> >>>>
> >>>>
> >>>>
> >>>>> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <[EMAIL PROTECTED]>
> wrote:
> >>>>>
> >>>>> Awesome Philip, thanks for sharing!
> >>>>>
> >>>>>> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <[EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>
> >>>>>> A couple of us here at Loggly recently spoke at AWS reinvent, on
> how we
> >>>>>> use Kafka 0.72 in our ingestion pipeline. The slides are at the link
> >>>>> below,
> >>>>>> and may be of interest to people on this list.
> >>>>
> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
> >>>>>>
> >>>>>> Any questions, let me know, though I can't promise I can answer
> >>>>>> everything. Can't give the complete game away. :-)
> >>>>>>
> >>>>>> As always, Kafka rocks!
> >>>>>>
> >>>>>> Philip
> >>>
> >>>
> >>>
> >>> --
> >>> ___________________________________________
> >>> Available at these partners:
> >>> [image: CloudFlare | shopify | Bigcommerce]
> >>>
> >>> SUREN HIRAMAN, VP TECHNOLOGY
> >>> SOCIOCAST
> >>> Simple. Powerful. Predictions.
> >>>
> >>> 96 SPRING STREET, 7TH FLOOR
> >>> NEW YORK, NY 10012
> >>> O: (917) 525-2466 ext. 105
> >>> F: 646.349.4063
> >>> E: [EMAIL PROTECTED]
> >>> W: www.sociocast.com
> >>>
> >>> Increase Conversion Rates up to 500%. Go to www.sociocast.com and
> enter
> >>> your URL for a free trial!
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB