Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Analysis of producer performance


Copy link to this message
-
Re: Analysis of producer performance
Piotr,

Not sure where the updated numbers are. but what you described may make
sense. In the no ack mode, if the broker is saturated, it will put back
pressure to the producer. Eventually, the producer will slow down because
the socket buffer is full. One big difference btw 0.8 and 0.7 is that the
0.8 broker has the overhead of recompressing compressed messages. If you
only have one partition in your test, all producers have to synchronize on
the log when doing recompressing, which could limit the throughput. To
improve the throughput, you can try using more partitions, turning off
compression or using a cheaper compression codec like snappy.

Thanks,

Jun
On Fri, Apr 12, 2013 at 4:08 PM, Piotr Kozikowski <[EMAIL PROTECTED]>wrote:

> Hi all,
>
> I posted an update on the post (
> https://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/) to
> test the effect of disabling ack messages from brokers. It appears this
> only makes a big difference (~2x improvement ) when using synthetic log
> messages, but only a modest 12% improvement when using real production
> messages. This is using GZIP compression. The way I interpret this is that
> just turning acks off is not enough to mimic the 0.7 behavior because GZIP
> consumes significant CPU time and since the brokers now need to decompress
> data, there is a hit on throughput even without acks. Does this sound
> reasonable?
>
> Thanks,
>
> Piotr
>
> On Mon, Apr 8, 2013 at 4:42 PM, Piotr Kozikowski <[EMAIL PROTECTED]>
> wrote:
>
> > Hi,
> >
> > At LiveRamp we are considering replacing Scribe with Kafka, and as a
> first
> > step we run some tests to evaluate producer performance. You can find our
> > preliminary results here:
> > https://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/.
> > We hope this will be useful for some folks, and If anyone has comments or
> > suggestions about what to do differently to obtain better results your
> > feedback will be very welcome.
> >
> > Thanks,
> >
> > Piotr
> >
>