Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Re: Abou Kafka 0.8 producer throughput test


Copy link to this message
-
Re: Abou Kafka 0.8 producer throughput test
Looks like Jun's email didn't format the output properly. I've published
some preliminary producer throughput performance numbers on our performance
wiki -
https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing#Performancetesting-Producerthroughput

These tests measure producer throughput in the worst case scenario
(producer.num.acks=-1) i.e. max durability setting. The baseline with 0.7
would be to compare producer throughput with num.acks=0. We are working on
those tests now.

Thanks,
Neha
On Thu, Jan 17, 2013 at 8:43 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> We also did some perf test on 0.8 using the following command. All configs
> on the broker are the defaults.
> bin/kafka-run-class.sh kafka.perf.ProducerPerformance --broker-list
> localhost:9092 --initial-message-id 0 --messages 2000000 --topics topic_001
> --request-num-acks -1 --batch-size 100 --threads 1 --message-size 1024
> --compression-codec 0
>
> The following is our preliminary result. Could you try this on your
> environment? For replication factor larger than 1, we will try ack=1 and
> report the numbers later. It should provide better throughput. Thanks,
>
> *No. of Brokers = 1 / Replication Factor = 1 (Partition = 1)**Producer
> threads**comp**msg size**Acks**batch**Thru Put
> (MB/s)*101024-115.49201024-11
> 9.38501024-1116.611001024-1119.54101024-15025.72201024-15039.25501024-150
>
> 54.171001024-15056.71101024-110027.97201024-110045.05501024-110058.011001024
> -110059.82*No. of Brokers = 2 / Replication Factor = 2 (Partitions =
> 1)**Producer
> threads**comp**msg size**Acks**batch**Thru Put
> (MB/s)*101024-110.58201024-11
> 1.17501024-111.601001024-113.15101024-1507.48201024-15013.89501024-15018.11
> 1001024-15020.91101024-11008.72201024-110016.84501024-110020.661001024-1100
> 23.82*No. of Brokers = 3 / Replication Factor = 3 (Partitions =
> 1)**Producer
> threads**comp**msg size**Acks**batch**Thru Put
> (MB/s)*101024-110.53201024-11
> 0.94501024-111.721001024-112.78101024-1507.08201024-15013.40501024-15018.11
> 1001024-15021.01101024-11008.09201024-110014.88501024-110019.931001024-1100
> 23.22
> Thanks,
>
> Jun
>
> On Wed, Jan 16, 2013 at 8:33 PM, Jun Guo -X (jungu - CIIC at Cisco) <
> [EMAIL PROTECTED]> wrote:
>
> >  Hi,****
> >
> >       I do producer(Kafka 0.8) throughput test many times. But the
> > average value is 3MB/S. Below is my test environment:****
> >
> >        CPU core      :16 ****
> >
> >        Vendor_id     :GenuineIntel****
> >
> >        Cpu family     :6****
> >
> >        Cpu MHz      :2899.999****
> >
> >        Cache size    :20480 KB****
> >
> >        Cpu level      :13****
> >
> >        MEM             :16330832KB=15.57GB****
> >
> >        Disk       : RAID5****
> >
> > ** **
> >
> >        I don’t know the detail information about the disk, such as
> > rotation. But I do some test about the I/O performance of the disk. The
> > write rate is 500MB~600MB/S, the read rate is 180MB/S. The detail is as
> > below. ****
> >
> > [image: cid:[EMAIL PROTECTED]6900]****
> >
> > ** **
> >
> > And I adjust the broker configuration file as the official document says
> > as below. And I adjust the JVM to 5120MB. ****
> >
> > I run producer performance test with the script
> > kafka-producer-perf-test.sh, with the test command is ****
> >
> > *bin/kafka-producer-perf-test.sh --broker-list 10.75.167.46:49092--topics
> topic_perf_46_1,topic_perf_46_2,topic_perf_46_3,topic_perf_46_4,
> > topic_perf_46_5,topic_perf_46_6,
> > topic_perf_46_7,topic_perf_46_8,topic_perf_46_9,topic_perf_46_10
> > --initial-message-id 0 --threads 200 --messages 1000000 --message-size
> 200
> > --compression-codec 1*
> >
> > ** **
> >
> > But the test result is also not as good as the official document
> > says(50MB/S, and that value in your paper is 100MB/S). The test result is
> > as below:****
> >
> > 2013-01-17 04:15:24:768, 2013-01-17 04:25:01:637, 0, 200, 200, 1907.35, *
> > 3.3064,* 10000000, 17334.9582****
> >
> > ** **
> >
> > On the other hand, I do consumer throughput test, the result is about

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB