Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> librdkafka 0.8.0 released


Copy link to this message
-
Re: librdkafka 0.8.0 released
Thanks. The results make sense. Higher consistency (ack=-1 and ack=2)
typically means longer latency.

Do those number match our java producers?

Thanks,

Jun
On Mon, Nov 25, 2013 at 1:16 PM, Magnus Edenhill <[EMAIL PROTECTED]> wrote:

> Producing to one partition, no replication, required.acks = 0:
> % 1000000 messages and 100000000 bytes produced in 1215ms: 822383 msgs/s
> and 82.24 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, no replication, required.acks = -1:
> % 1000000 messages and 100000000 bytes produced in 1422ms: 703091 msgs/s
> and 70.31 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, no replication, required.acks = 1:
> % 1000000 messages and 100000000 bytes produced in 1295ms: 771881 msgs/s
> and 77.19 Mb/s, 0 messages failed, no compression
>
>
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = 0:
> % 1000000 messages and 100000000 bytes produced in 1354ms: 738483 msgs/s
> and 73.85 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = -1:
> % 1000000 messages and 100000000 bytes produced in 3698ms: 270396 msgs/s
> and 27.04 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = 1:
> % 1000000 messages and 100000000 bytes produced in 1360ms: 735224 msgs/s
> and 73.52 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = 2:
> % 1000000 messages and 100000000 bytes produced in 3568ms: 280241 msgs/s
> and 28.02 Mb/s, 0 messages failed, no compression
>
>
> These are the maximum values from a smaller number of naive tests.
>
> It would be interesting to see some numbers from relevant environments with
> proper hardware and networks.
> (rdkafka_performance -P -t <topic> -p <partition> -s <msgsize> -a
> <required.acks> -c 1000000 -q)
>
> Regards,
> Magnus
>
>
> 2013/11/25 Jun Rao <[EMAIL PROTECTED]>
>
> > Thanks for sharing the results. Was the topic created with replication
> > factor of 2? Could you test acks=-1 as well?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Nov 25, 2013 at 4:30 AM, Magnus Edenhill <[EMAIL PROTECTED]>
> > wrote:
> >
> > > The following tests were using a single producer application
> > > (examples/rdkafka_performance):
> > >
> > > * Test1: 2 brokers, 2 partitions, required.acks=2, 100 byte messages:
> > > 850000 messages/second, 85 MB/second
> > >
> > > * Test2: 1 broker, 1 partition, required.acks=0, 100 byte messages:
> > 710000
> > > messages/second, 71 MB/second
> > >
> > > * Test3: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
> > > snappy compression: 300000 messages/second, 30 MB/second
> > >
> > > * Test4: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
> > gzip
> > > compression: 230000 messages/second, 23 MB/second
> > >
> > >
> > > log.flush broker configuration was increased to avoid the disk being
> the
> > > bottleneck.
> > >
> > >
> > > /Magnus
> > >
> > >
> > >
> > > 2013/11/24 Neha Narkhede <[EMAIL PROTECTED]>
> > >
> > > > So, a single producer'a throughput is 80 MB/s? That seems pretty
> high.
> > > What
> > > > was the number of acks setting? Thanks for sharing these numbers.
> > > >
> > > > On Sunday, November 24, 2013, Magnus Edenhill wrote:
> > > >
> > > > > Hi Neha,
> > > > >
> > > > > these tests were done using 100 byte messages. More information
> about
> > > the
> > > > > producer performance tests can be found here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > > > >
> > > > > The tests are indicative at best and in no way scientific, but I
> must
> > > say
> > > > > that the Kafka broker performance is impressive.
> > > > >
> > > > > Regards,
> > > > > Magnus
> > > > >
> > > > >
> > > > >
> > > > > 2013/11/22 Neha Narkhede <[EMAIL PROTECTED] <javascript:;>>