Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka brokers in a VM


Copy link to this message
-
Re: Kafka brokers in a VM
It's a question we're asking ourselves at the moment too :-), was hoping
you might have had some insight.

Out of curiosity - How are your EBS drives set up? Are they on a raid-0?

On Thu, Aug 9, 2012 at 6:55 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> @Matthew - that's an interesting idea, we haven't explored it yet.
> I'm sure it would be better, but then some backup option is needed or
> you have to be OK with losing the data, right?   Or do you rely on
> replication or something?
>
> On Thu, Aug 9, 2012 at 4:47 PM, Matthew Rathbone <[EMAIL PROTECTED]>
> wrote:
> > @Evan -- how does performance vary if you use the ephemeral drives
> instead
> > of EBS?
> >
> > On Thu, Aug 9, 2012 at 6:41 PM, Evan Chan <[EMAIL PROTECTED]> wrote:
> >
> >> We run Kafka instances on AWS and I had posted performance #'s before.
> >>  We achieve around 20MB/sec IO throughput, with the Kafka messages
> >> stored in EBS, using reiserfs.  I wonder how much faster we could get
> >> though, using 1) ext4, and 2) SSDs.
> >>
> >> -Evan
> >>
> >>
> >> On Thu, Aug 9, 2012 at 11:25 AM, Erich Nachbar <[EMAIL PROTECTED]>
> wrote:
> >> > We are running 3 Kafka instances on Proxmox (container based
> >> > virtualization, with typical 1-5% overhead) in production.
> >> >
> >> > While I have not benchmarked Kafka on it, other systems I performance
> >> > tested before on Proxmox showed no measurable performance difference.
> It
> >> > runs OpenVZ under the hood for its containers, which is loved by all
> the
> >> > cheap $15/mo hosters for its efficiency.
> >> >
> >> > On Thu, Aug 9, 2012 at 8:16 AM, Jay Kreps <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> >> In the past I had used AWS for some performance testing. I didn't see
> >> >> anything too unexpected other than the usual overhead on first write
> and
> >> >> general mediocre AWS network performance.
> >> >>
> >> >> -jay
> >> >>
> >> >> On Wed, Aug 8, 2012 at 11:22 AM, Casey Sybrandy <
> >> >> [EMAIL PROTECTED]> wrote:
> >> >>
> >> >> > Hello,
> >> >> >
> >> >> > Has anyone deployed their kafka brokers within a VM?  Are their any
> >> >> > performance issues?  I'd expect that once we start hitting it hard,
> >> the
> >> >> > disk writes will be an issue, however I wanted to get some
> additional
> >> >> > information.
> >> >> >
> >> >> > Thanks.
> >> >> >
> >> >> > Casey
> >> >> >
> >> >>
> >>
> >>
> >>
> >> --
> >> --
> >> Evan Chan
> >> Senior Software Engineer |
> >> [EMAIL PROTECTED] | (650) 996-4600
> >> www.ooyala.com | blog | @ooyala
> >>
> >
> >
> >
> > --
> > Matthew Rathbone
> > Foursquare | Software Engineer | Server Engineering Team
> > [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> > 4sq<http://foursquare.com/rathboma>
>
>
>
> --
> --
> Evan Chan
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog | @ooyala
>

--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>