Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka brokers in a VM


Copy link to this message
-
Re: Kafka brokers in a VM
@Evan -- how does performance vary if you use the ephemeral drives instead
of EBS?

On Thu, Aug 9, 2012 at 6:41 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> We run Kafka instances on AWS and I had posted performance #'s before.
>  We achieve around 20MB/sec IO throughput, with the Kafka messages
> stored in EBS, using reiserfs.  I wonder how much faster we could get
> though, using 1) ext4, and 2) SSDs.
>
> -Evan
>
>
> On Thu, Aug 9, 2012 at 11:25 AM, Erich Nachbar <[EMAIL PROTECTED]> wrote:
> > We are running 3 Kafka instances on Proxmox (container based
> > virtualization, with typical 1-5% overhead) in production.
> >
> > While I have not benchmarked Kafka on it, other systems I performance
> > tested before on Proxmox showed no measurable performance difference. It
> > runs OpenVZ under the hood for its containers, which is loved by all the
> > cheap $15/mo hosters for its efficiency.
> >
> > On Thu, Aug 9, 2012 at 8:16 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> In the past I had used AWS for some performance testing. I didn't see
> >> anything too unexpected other than the usual overhead on first write and
> >> general mediocre AWS network performance.
> >>
> >> -jay
> >>
> >> On Wed, Aug 8, 2012 at 11:22 AM, Casey Sybrandy <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >> > Hello,
> >> >
> >> > Has anyone deployed their kafka brokers within a VM?  Are their any
> >> > performance issues?  I'd expect that once we start hitting it hard,
> the
> >> > disk writes will be an issue, however I wanted to get some additional
> >> > information.
> >> >
> >> > Thanks.
> >> >
> >> > Casey
> >> >
> >>
>
>
>
> --
> --
> Evan Chan
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog | @ooyala
>

--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB