Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka brokers in a VM


Copy link to this message
-
Re: Kafka brokers in a VM
It's a question we're asking ourselves at the moment too :-), was hoping
you might have had some insight.

Out of curiosity - How are your EBS drives set up? Are they on a raid-0?

On Thu, Aug 9, 2012 at 6:55 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> @Matthew - that's an interesting idea, we haven't explored it yet.
> I'm sure it would be better, but then some backup option is needed or
> you have to be OK with losing the data, right?   Or do you rely on
> replication or something?
>
> On Thu, Aug 9, 2012 at 4:47 PM, Matthew Rathbone <[EMAIL PROTECTED]>
> wrote:
> > @Evan -- how does performance vary if you use the ephemeral drives
> instead
> > of EBS?
> >
> > On Thu, Aug 9, 2012 at 6:41 PM, Evan Chan <[EMAIL PROTECTED]> wrote:
> >
> >> We run Kafka instances on AWS and I had posted performance #'s before.
> >>  We achieve around 20MB/sec IO throughput, with the Kafka messages
> >> stored in EBS, using reiserfs.  I wonder how much faster we could get
> >> though, using 1) ext4, and 2) SSDs.
> >>
> >> -Evan
> >>
> >>
> >> On Thu, Aug 9, 2012 at 11:25 AM, Erich Nachbar <[EMAIL PROTECTED]>
> wrote:
> >> > We are running 3 Kafka instances on Proxmox (container based
> >> > virtualization, with typical 1-5% overhead) in production.
> >> >
> >> > While I have not benchmarked Kafka on it, other systems I performance
> >> > tested before on Proxmox showed no measurable performance difference.
> It
> >> > runs OpenVZ under the hood for its containers, which is loved by all
> the
> >> > cheap $15/mo hosters for its efficiency.
> >> >
> >> > On Thu, Aug 9, 2012 at 8:16 AM, Jay Kreps <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> >> In the past I had used AWS for some performance testing. I didn't see
> >> >> anything too unexpected other than the usual overhead on first write
> and
> >> >> general mediocre AWS network performance.
> >> >>
> >> >> -jay
> >> >>
> >> >> On Wed, Aug 8, 2012 at 11:22 AM, Casey Sybrandy <
> >> >> [EMAIL PROTECTED]> wrote:
> >> >>
> >> >> > Hello,
> >> >> >
> >> >> > Has anyone deployed their kafka brokers within a VM?  Are their any
> >> >> > performance issues?  I'd expect that once we start hitting it hard,
> >> the
> >> >> > disk writes will be an issue, however I wanted to get some
> additional
> >> >> > information.
> >> >> >
> >> >> > Thanks.
> >> >> >
> >> >> > Casey
> >> >> >
> >> >>
> >>
> >>
> >>
> >> --
> >> --
> >> Evan Chan
> >> Senior Software Engineer |
> >> [EMAIL PROTECTED] | (650) 996-4600
> >> www.ooyala.com | blog | @ooyala
> >>
> >
> >
> >
> > --
> > Matthew Rathbone
> > Foursquare | Software Engineer | Server Engineering Team
> > [EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
> > 4sq<http://foursquare.com/rathboma>
>
>
>
> --
> --
> Evan Chan
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog | @ooyala
>

--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB