Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> librdkafka 0.8.0 released


Copy link to this message
-
Re: librdkafka 0.8.0 released
Thanks. The results make sense. Higher consistency (ack=-1 and ack=2)
typically means longer latency.

Do those number match our java producers?

Thanks,

Jun
On Mon, Nov 25, 2013 at 1:16 PM, Magnus Edenhill <[EMAIL PROTECTED]> wrote:

> Producing to one partition, no replication, required.acks = 0:
> % 1000000 messages and 100000000 bytes produced in 1215ms: 822383 msgs/s
> and 82.24 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, no replication, required.acks = -1:
> % 1000000 messages and 100000000 bytes produced in 1422ms: 703091 msgs/s
> and 70.31 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, no replication, required.acks = 1:
> % 1000000 messages and 100000000 bytes produced in 1295ms: 771881 msgs/s
> and 77.19 Mb/s, 0 messages failed, no compression
>
>
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = 0:
> % 1000000 messages and 100000000 bytes produced in 1354ms: 738483 msgs/s
> and 73.85 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = -1:
> % 1000000 messages and 100000000 bytes produced in 3698ms: 270396 msgs/s
> and 27.04 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = 1:
> % 1000000 messages and 100000000 bytes produced in 1360ms: 735224 msgs/s
> and 73.52 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = 2:
> % 1000000 messages and 100000000 bytes produced in 3568ms: 280241 msgs/s
> and 28.02 Mb/s, 0 messages failed, no compression
>
>
> These are the maximum values from a smaller number of naive tests.
>
> It would be interesting to see some numbers from relevant environments with
> proper hardware and networks.
> (rdkafka_performance -P -t <topic> -p <partition> -s <msgsize> -a
> <required.acks> -c 1000000 -q)
>
> Regards,
> Magnus
>
>
> 2013/11/25 Jun Rao <[EMAIL PROTECTED]>
>
> > Thanks for sharing the results. Was the topic created with replication
> > factor of 2? Could you test acks=-1 as well?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Nov 25, 2013 at 4:30 AM, Magnus Edenhill <[EMAIL PROTECTED]>
> > wrote:
> >
> > > The following tests were using a single producer application
> > > (examples/rdkafka_performance):
> > >
> > > * Test1: 2 brokers, 2 partitions, required.acks=2, 100 byte messages:
> > > 850000 messages/second, 85 MB/second
> > >
> > > * Test2: 1 broker, 1 partition, required.acks=0, 100 byte messages:
> > 710000
> > > messages/second, 71 MB/second
> > >
> > > * Test3: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
> > > snappy compression: 300000 messages/second, 30 MB/second
> > >
> > > * Test4: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
> > gzip
> > > compression: 230000 messages/second, 23 MB/second
> > >
> > >
> > > log.flush broker configuration was increased to avoid the disk being
> the
> > > bottleneck.
> > >
> > >
> > > /Magnus
> > >
> > >
> > >
> > > 2013/11/24 Neha Narkhede <[EMAIL PROTECTED]>
> > >
> > > > So, a single producer'a throughput is 80 MB/s? That seems pretty
> high.
> > > What
> > > > was the number of acks setting? Thanks for sharing these numbers.
> > > >
> > > > On Sunday, November 24, 2013, Magnus Edenhill wrote:
> > > >
> > > > > Hi Neha,
> > > > >
> > > > > these tests were done using 100 byte messages. More information
> about
> > > the
> > > > > producer performance tests can be found here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > > > >
> > > > > The tests are indicative at best and in no way scientific, but I
> must
> > > say
> > > > > that the Kafka broker performance is impressive.
> > > > >
> > > > > Regards,
> > > > > Magnus
> > > > >
> > > > >
> > > > >
> > > > > 2013/11/22 Neha Narkhede <[EMAIL PROTECTED] <javascript:;>>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB