Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Re: Abou Kafka 0.8 producer throughput test


Copy link to this message
-
Re: Abou Kafka 0.8 producer throughput test
S Ahmed 2013-01-18, 20:25
I see ok, so if you wanted to compare .7 with .8 on the same footing, then
you would set it to 0 right? (since 0.7 is fire and forget)

producer.num.acks=0
On Thu, Jan 17, 2013 at 11:45 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> I means wait for the data reaches all replicas (that are in sync).
>
> Thanks,
>
> Jun
>
> On Thu, Jan 17, 2013 at 6:42 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
>
> > producer.num.acks=-1 means what sorry? is it that all replica's are
> written
> > too?
> >
> >
> > On Thu, Jan 17, 2013 at 12:09 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Looks like Jun's email didn't format the output properly. I've
> published
> > > some preliminary producer throughput performance numbers on our
> > performance
> > > wiki -
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing#Performancetesting-Producerthroughput
> > >
> > > These tests measure producer throughput in the worst case scenario
> > > (producer.num.acks=-1) i.e. max durability setting. The baseline with
> 0.7
> > > would be to compare producer throughput with num.acks=0. We are working
> > on
> > > those tests now.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Jan 17, 2013 at 8:43 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > We also did some perf test on 0.8 using the following command. All
> > > configs
> > > > on the broker are the defaults.
> > > > bin/kafka-run-class.sh kafka.perf.ProducerPerformance --broker-list
> > > > localhost:9092 --initial-message-id 0 --messages 2000000 --topics
> > > topic_001
> > > > --request-num-acks -1 --batch-size 100 --threads 1 --message-size
> 1024
> > > > --compression-codec 0
> > > >
> > > > The following is our preliminary result. Could you try this on your
> > > > environment? For replication factor larger than 1, we will try ack=1
> > and
> > > > report the numbers later. It should provide better throughput.
> Thanks,
> > > >
> > > > *No. of Brokers = 1 / Replication Factor = 1 (Partition =
> 1)**Producer
> > > > threads**comp**msg size**Acks**batch**Thru Put
> > > > (MB/s)*101024-115.49201024-11
> > > >
> > 9.38501024-1116.611001024-1119.54101024-15025.72201024-15039.25501024-150
> > > >
> > > >
> > >
> >
> 54.171001024-15056.71101024-110027.97201024-110045.05501024-110058.011001024
> > > > -110059.82*No. of Brokers = 2 / Replication Factor = 2 (Partitions =
> > > > 1)**Producer
> > > > threads**comp**msg size**Acks**batch**Thru Put
> > > > (MB/s)*101024-110.58201024-11
> > > >
> > >
> >
> 1.17501024-111.601001024-113.15101024-1507.48201024-15013.89501024-15018.11
> > > >
> > >
> >
> 1001024-15020.91101024-11008.72201024-110016.84501024-110020.661001024-1100
> > > > 23.82*No. of Brokers = 3 / Replication Factor = 3 (Partitions =
> > > > 1)**Producer
> > > > threads**comp**msg size**Acks**batch**Thru Put
> > > > (MB/s)*101024-110.53201024-11
> > > >
> > >
> >
> 0.94501024-111.721001024-112.78101024-1507.08201024-15013.40501024-15018.11
> > > >
> > >
> >
> 1001024-15021.01101024-11008.09201024-110014.88501024-110019.931001024-1100
> > > > 23.22
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jan 16, 2013 at 8:33 PM, Jun Guo -X (jungu - CIIC at Cisco) <
> > > > [EMAIL PROTECTED]> wrote:
> > > >
> > > > >  Hi,****
> > > > >
> > > > >       I do producer(Kafka 0.8) throughput test many times. But the
> > > > > average value is 3MB/S. Below is my test environment:****
> > > > >
> > > > >        CPU core      :16 ****
> > > > >
> > > > >        Vendor_id     :GenuineIntel****
> > > > >
> > > > >        Cpu family     :6****
> > > > >
> > > > >        Cpu MHz      :2899.999****
> > > > >
> > > > >        Cache size    :20480 KB****
> > > > >
> > > > >        Cpu level      :13****
> > > > >
> > > > >        MEM             :16330832KB=15.57GB****
> > > > >
> > > > >        Disk       : RAID5****
> > > > >
> > > > > ** **
> > > > >
> > > > >        I don’t know the detail information about the disk, such as
> > > > > rotation. But I do some test about the I/O performance of the disk.