Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Re: Abou Kafka 0.8 producer throughput test


+
Jun Rao 2013-01-17, 16:43
+
Neha Narkhede 2013-01-17, 17:09
Copy link to this message
-
Re: Abou Kafka 0.8 producer throughput test
producer.num.acks=-1 means what sorry? is it that all replica's are written
too?
On Thu, Jan 17, 2013 at 12:09 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Looks like Jun's email didn't format the output properly. I've published
> some preliminary producer throughput performance numbers on our performance
> wiki -
>
> https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing#Performancetesting-Producerthroughput
>
> These tests measure producer throughput in the worst case scenario
> (producer.num.acks=-1) i.e. max durability setting. The baseline with 0.7
> would be to compare producer throughput with num.acks=0. We are working on
> those tests now.
>
> Thanks,
> Neha
>
>
> On Thu, Jan 17, 2013 at 8:43 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > We also did some perf test on 0.8 using the following command. All
> configs
> > on the broker are the defaults.
> > bin/kafka-run-class.sh kafka.perf.ProducerPerformance --broker-list
> > localhost:9092 --initial-message-id 0 --messages 2000000 --topics
> topic_001
> > --request-num-acks -1 --batch-size 100 --threads 1 --message-size 1024
> > --compression-codec 0
> >
> > The following is our preliminary result. Could you try this on your
> > environment? For replication factor larger than 1, we will try ack=1 and
> > report the numbers later. It should provide better throughput. Thanks,
> >
> > *No. of Brokers = 1 / Replication Factor = 1 (Partition = 1)**Producer
> > threads**comp**msg size**Acks**batch**Thru Put
> > (MB/s)*101024-115.49201024-11
> > 9.38501024-1116.611001024-1119.54101024-15025.72201024-15039.25501024-150
> >
> >
> 54.171001024-15056.71101024-110027.97201024-110045.05501024-110058.011001024
> > -110059.82*No. of Brokers = 2 / Replication Factor = 2 (Partitions =
> > 1)**Producer
> > threads**comp**msg size**Acks**batch**Thru Put
> > (MB/s)*101024-110.58201024-11
> >
> 1.17501024-111.601001024-113.15101024-1507.48201024-15013.89501024-15018.11
> >
> 1001024-15020.91101024-11008.72201024-110016.84501024-110020.661001024-1100
> > 23.82*No. of Brokers = 3 / Replication Factor = 3 (Partitions =
> > 1)**Producer
> > threads**comp**msg size**Acks**batch**Thru Put
> > (MB/s)*101024-110.53201024-11
> >
> 0.94501024-111.721001024-112.78101024-1507.08201024-15013.40501024-15018.11
> >
> 1001024-15021.01101024-11008.09201024-110014.88501024-110019.931001024-1100
> > 23.22
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jan 16, 2013 at 8:33 PM, Jun Guo -X (jungu - CIIC at Cisco) <
> > [EMAIL PROTECTED]> wrote:
> >
> > >  Hi,****
> > >
> > >       I do producer(Kafka 0.8) throughput test many times. But the
> > > average value is 3MB/S. Below is my test environment:****
> > >
> > >        CPU core      :16 ****
> > >
> > >        Vendor_id     :GenuineIntel****
> > >
> > >        Cpu family     :6****
> > >
> > >        Cpu MHz      :2899.999****
> > >
> > >        Cache size    :20480 KB****
> > >
> > >        Cpu level      :13****
> > >
> > >        MEM             :16330832KB=15.57GB****
> > >
> > >        Disk       : RAID5****
> > >
> > > ** **
> > >
> > >        I don’t know the detail information about the disk, such as
> > > rotation. But I do some test about the I/O performance of the disk. The
> > > write rate is 500MB~600MB/S, the read rate is 180MB/S. The detail is as
> > > below. ****
> > >
> > > [image: cid:[EMAIL PROTECTED]6900]****
> > >
> > > ** **
> > >
> > > And I adjust the broker configuration file as the official document
> says
> > > as below. And I adjust the JVM to 5120MB. ****
> > >
> > > I run producer performance test with the script
> > > kafka-producer-perf-test.sh, with the test command is ****
> > >
> > > *bin/kafka-producer-perf-test.sh --broker-list 10.75.167.46:49092
> --topics
> > topic_perf_46_1,topic_perf_46_2,topic_perf_46_3,topic_perf_46_4,
> > > topic_perf_46_5,topic_perf_46_6,
> > > topic_perf_46_7,topic_perf_46_8,topic_perf_46_9,topic_perf_46_10
> > > --initial-message-id 0 --threads 200 --messages 1000000 --message-size

 
+
Jun Rao 2013-01-18, 04:45
+
S Ahmed 2013-01-18, 20:25
+
Neha Narkhede 2013-01-19, 01:43
+
S Ahmed 2013-01-23, 03:11
+
Jay Kreps 2013-01-23, 04:01
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB