Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Re: Abou Kafka 0.8 producer throughput test


Copy link to this message
-
Re: Abou Kafka 0.8 producer throughput test
We also did some perf test on 0.8 using the following command. All configs
on the broker are the defaults.
bin/kafka-run-class.sh kafka.perf.ProducerPerformance --broker-list
localhost:9092 --initial-message-id 0 --messages 2000000 --topics topic_001
--request-num-acks -1 --batch-size 100 --threads 1 --message-size 1024
--compression-codec 0

The following is our preliminary result. Could you try this on your
environment? For replication factor larger than 1, we will try ack=1 and
report the numbers later. It should provide better throughput. Thanks,

*No. of Brokers = 1 / Replication Factor = 1 (Partition = 1)**Producer
threads**comp**msg size**Acks**batch**Thru Put (MB/s)*101024-115.49201024-11
9.38501024-1116.611001024-1119.54101024-15025.72201024-15039.25501024-150
54.171001024-15056.71101024-110027.97201024-110045.05501024-110058.011001024
-110059.82*No. of Brokers = 2 / Replication Factor = 2 (Partitions =
1)**Producer
threads**comp**msg size**Acks**batch**Thru Put (MB/s)*101024-110.58201024-11
1.17501024-111.601001024-113.15101024-1507.48201024-15013.89501024-15018.11
1001024-15020.91101024-11008.72201024-110016.84501024-110020.661001024-1100
23.82*No. of Brokers = 3 / Replication Factor = 3 (Partitions = 1)**Producer
threads**comp**msg size**Acks**batch**Thru Put (MB/s)*101024-110.53201024-11
0.94501024-111.721001024-112.78101024-1507.08201024-15013.40501024-15018.11
1001024-15021.01101024-11008.09201024-110014.88501024-110019.931001024-1100
23.22
Thanks,

Jun

On Wed, Jan 16, 2013 at 8:33 PM, Jun Guo -X (jungu - CIIC at Cisco) <
[EMAIL PROTECTED]> wrote:

>  Hi,****
>
>       I do producer(Kafka 0.8) throughput test many times. But the
> average value is 3MB/S. Below is my test environment:****
>
>        CPU core      :16 ****
>
>        Vendor_id     :GenuineIntel****
>
>        Cpu family     :6****
>
>        Cpu MHz      :2899.999****
>
>        Cache size    :20480 KB****
>
>        Cpu level      :13****
>
>        MEM             :16330832KB=15.57GB****
>
>        Disk       : RAID5****
>
> ** **
>
>        I don’t know the detail information about the disk, such as
> rotation. But I do some test about the I/O performance of the disk. The
> write rate is 500MB~600MB/S, the read rate is 180MB/S. The detail is as
> below. ****
>
> [image: cid:[EMAIL PROTECTED]6900]****
>
> ** **
>
> And I adjust the broker configuration file as the official document says
> as below. And I adjust the JVM to 5120MB. ****
>
> I run producer performance test with the script
> kafka-producer-perf-test.sh, with the test command is ****
>
> *bin/kafka-producer-perf-test.sh --broker-list 10.75.167.46:49092--topics topic_perf_46_1,topic_perf_46_2,topic_perf_46_3,topic_perf_46_4,
> topic_perf_46_5,topic_perf_46_6,
> topic_perf_46_7,topic_perf_46_8,topic_perf_46_9,topic_perf_46_10
> --initial-message-id 0 --threads 200 --messages 1000000 --message-size 200
> --compression-codec 1*
>
> ** **
>
> But the test result is also not as good as the official document
> says(50MB/S, and that value in your paper is 100MB/S). The test result is
> as below:****
>
> 2013-01-17 04:15:24:768, 2013-01-17 04:25:01:637, 0, 200, 200, 1907.35, *
> 3.3064,* 10000000, 17334.9582****
>
> ** **
>
> On the other hand, I do consumer throughput test, the result is about
> 60MB/S while that value in official document is 100MB/S.****
>
> I really don’t know why?****
>
> You know high throughput is one of the most important features of Kafka.
> So I am really concerned with it.****
>
> ** **
>
> Thanks and best regards!****
>
> ** **
>
> *From:* Jay Kreps [mailto:[EMAIL PROTECTED]]
> *Sent:* 2013年1月16日 2:22
> *To:* Jun Guo -X (jungu - CIIC at Cisco)
> *Subject:* RE: About acknowledge from broker to producer in your paper.***
> *
>
> ** **
>
> Not sure which version you are using... ****
>
> ** **
>
> In 0.7 this would happen only if there was a socket level error (i.e.
> can't connect to the host). This covers a lot of cases since in the event
> of I/O errors (disk full, etc) we just have that node shut itself down to

 
+
Neha Narkhede 2013-01-17, 17:09
+
S Ahmed 2013-01-18, 02:43
+
Jun Rao 2013-01-18, 04:45
+
S Ahmed 2013-01-18, 20:25
+
Neha Narkhede 2013-01-19, 01:43
+
S Ahmed 2013-01-23, 03:11
+
Jay Kreps 2013-01-23, 04:01