Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Producer reaches a max of 7Mbps


Copy link to this message
-
Re: Producer reaches a max of 7Mbps
Neha Narkhede 2013-11-19, 14:27
I went through the performance page where  it can reach a speed of 50MBps.

I think that number is true for 07, not 08. If you want higher producer
throughout in 08, you can set request . required.acks=0. Note that it means
that the producer does not receive server side acknowledgements if you use
that config. We plan to address the 08 throughput issue in 09.

Thanks,
Neha
On Nov 18, 2013 11:50 PM, "Abhinav Anand" <[EMAIL PROTECTED]> wrote:

> Hi,
>  I am using kafka producer and broker for a production setup. The expected
> producer output is 20MBps but I am only getting max of 8MBps. I have
> verified that we are losing packets by directly connecting to the data
> source through TCP though the metrics is not reflecting any loss.
> I went through the performance page where  it can reach a speed of 50MBps.
> Please look at the config and suggest if there is some configuration
> improvement i can do.
>
> *** *Message Size* ***
> Message size = 3KB
>
> ***** Producer Config ****
> producer.type = async
> queue.buffering.max.ms = 100
> queue.buffering.max.messages = 4000
> request.timeout.ms = 30000
> batch.num.messages = 200
>
> **** Broker Config* ***
>
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=1048576
> socket.receive.buffer.bytes=2097152
> socket.request.max.bytes=104857600
> log.dir=/data1/kafka/logs
> num.partitions=1
> log.flush.interval.messages=1000
> log.flush.interval.ms=300
> log.retention.hours=48
> log.retention.bytes=107374182400
> log.segment.bytes=536870912
> log.cleanup.interval.mins=1
> zookeeper.connect=dare-msgq00:2181,dare-msgq01:2181,dare-msgq02:2181
> zookeeper.connection.timeout.ms=1000000
>
> --
> Abhinav Anand
>