Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka on EC2


Copy link to this message
-
Re: Kafka on EC2
Matthew Rathbone 2012-11-19, 21:44
If you raid EBS volumes you're pretty much protected from data loss as an
added benefit. So yeah if a broker goes down, you don't have the data until
you bring it back up, but you CAN bring it back up, even if the box dies.
On Mon, Nov 19, 2012 at 3:34 PM, Matt Jones <[EMAIL PROTECTED]> wrote:

> We have pretty much the same setup as this, running on m1.large instances.
>
> On Mon, Nov 19, 2012 at 11:44 AM, Matthew Rathbone
> <[EMAIL PROTECTED]>wrote:
>
> > We RAID-0 4 EBS disks, we find that to be most performant, although it
> does
> > leave you more vulnerable to EBS network errors and outages. Ideally, if
> > you're spread across AZ's you could do some clever routing to
> > geographically local brokers, whilst keeping the others as a backup in
> case
> > of failure.
> >
> > In the same vein, we have N+1 brokers, where N is how many we reasonably
> > think we need, this way we can hopefully survive outages.
> >
> >
> > On Mon, Nov 19, 2012 at 1:30 PM, David Arthur <[EMAIL PROTECTED]> wrote:
> >
> > > I'd only consider m1.xlarge and higher for Kafka. The m1.xlarge have
> > > "high" I/O performance according to Amazon. This is disk I/O and
> network
> > > I/O performance. Of course you need to use EBS volumes if you want your
> > > Kafka brokers to survive reboots - you can expect reboots on AWS. Some
> > > people have reported I/O improvements by RAIDing EBS volumes (
> > > http://alestic.com/2009/06/ec2-ebs-raid). Deploying in the same region
> > as
> > > your application will also improve performance.
> > >
> > >
> > > On Nov 16, 2012, at 8:04 PM, Senthilvel Rangaswamy wrote:
> > >
> > > > Have folks implemented large installations on Kafka on Amazon EC2. I
> am
> > > > looking for best practices. Like the kind of nodes, EBS vs Instance
> > store
> > > > etc.,
> > > >
> > > > --
> > > > ..Senthil
> > > >
> > > > "If there's anything more important than my ego around, I want it
> > > > caught and shot now."
> > > >                                                    - Douglas Adams.
> > >
> > >
> >
> >
> > --
> > Matthew Rathbone
> > Foursquare | Software Engineer | Server Engineering Team
> > [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> > 4sq<http://foursquare.com/rathboma>
> >
>

--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>