Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka brokers in a VM


Copy link to this message
-
Re: Kafka brokers in a VM
Matthew Rathbone 2012-08-09, 23:47
@Evan -- how does performance vary if you use the ephemeral drives instead
of EBS?

On Thu, Aug 9, 2012 at 6:41 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> We run Kafka instances on AWS and I had posted performance #'s before.
>  We achieve around 20MB/sec IO throughput, with the Kafka messages
> stored in EBS, using reiserfs.  I wonder how much faster we could get
> though, using 1) ext4, and 2) SSDs.
>
> -Evan
>
>
> On Thu, Aug 9, 2012 at 11:25 AM, Erich Nachbar <[EMAIL PROTECTED]> wrote:
> > We are running 3 Kafka instances on Proxmox (container based
> > virtualization, with typical 1-5% overhead) in production.
> >
> > While I have not benchmarked Kafka on it, other systems I performance
> > tested before on Proxmox showed no measurable performance difference. It
> > runs OpenVZ under the hood for its containers, which is loved by all the
> > cheap $15/mo hosters for its efficiency.
> >
> > On Thu, Aug 9, 2012 at 8:16 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> In the past I had used AWS for some performance testing. I didn't see
> >> anything too unexpected other than the usual overhead on first write and
> >> general mediocre AWS network performance.
> >>
> >> -jay
> >>
> >> On Wed, Aug 8, 2012 at 11:22 AM, Casey Sybrandy <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >> > Hello,
> >> >
> >> > Has anyone deployed their kafka brokers within a VM?  Are their any
> >> > performance issues?  I'd expect that once we start hitting it hard,
> the
> >> > disk writes will be an issue, however I wanted to get some additional
> >> > information.
> >> >
> >> > Thanks.
> >> >
> >> > Casey
> >> >
> >>
>
>
>
> --
> --
> Evan Chan
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog | @ooyala
>

--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>