Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka 0.7 performance compared to bare metal


Copy link to this message
-
Re: Kafka 0.7 performance compared to bare metal
Jun Rao 2013-08-30, 14:38
Are the clients on the same host as the broker? Could network be the
bottleneck?

Thanks,

Jun
On Fri, Aug 30, 2013 at 1:34 AM, Rafael Bagmanov <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am trying to understand how fast is kafka 0.7 compared to what I can get
> from hard drive. In essence I have 3 questions.
>
> In all tests below, I'm using single broker with single one-partitioned
> topic. Kafka perf tests have been run in 2 deployment configs:
> - broker, perf-test on same host
> - broker, perf-test on different hosts (the results are practically the
> same, so wont post them here)
>
>
> I'm using FIO(http://freecode.com/projects/fio) to benchmark speed of hard
> drives.
>
> Hardware I'm using:
> 1) m1.xlarge with ephemeral storage, 4 core cpu, 16 GB ram
> 2) hi1.4xlarge  with SSD, 16 core cpu, 64 GB ram
> 3) desktop machine with 7200 rpm sata, 4 core cpu, 8 GB ram
>
> Kafka broker config:
> Oracle jdk 1.6.0_38,  -Xmx2048
>
> socket.send.buffer=16777216
> socket.receive.buffer=16777216
> max.socket.request.bytes=104857600
> log.flush.interval=10000
> log.default.flush.interval.ms=1000
> log.default.flush.scheduler.interval.ms=1000
> num.threads=[num of cores]
>
>
> For kafka-producer-perf-test I'm assuming that IO access pattern is
> sequential write.
>
> Here is the test I ran with FIO:
>
> [sequential-write]
> rw=write
> size=50G
> ioengine=sync
> numjobs=1
> directory=/tmp/fio
> filename=redo01.log
>
>
> Here is kafka performance test:
>
> ./bin/kafka-producer-perf-test.sh -topic "perf" --batch-size 3000
> --messages 50000000 --message-size 1300 --brokerinfo
> broker.list=0:host:9092 --threads [number-of-cores]
>
>
> ----------------------------------------------------------------------------------------
> |           |   m1.xlarge            |    hi1.4xlarge       |  desktop
>  |
>
>  ----------------------------------------------------------------------------------------
> |  kafka  |     41 MB/s           |      217 MB/s       |     42 MB/s   |
>
>  -----------------------------------------------------------------------------------------
> |  fio      |     106 MB/s          |      377 MB/s       |    74 MB/s   |
>
> ----------------------------------------------------------------------------------------
>
>
> Question 1: The proportion (~1/2) is pretty stable against different kind
> of hardware I've tried. Is it as expected? Can something be done to improve
> this?
>
> I've tried to play with:
> log.flush.interval=10000
> log.default.flush.interval.ms=1000
> log.default.flush.scheduler.interval.ms=1000
>
> Like increasing 10 times, or decreasing 10 times, but haven't seen much of
> a difference in IO  throughput
>
> The other thing that bugs me much more is that kafka consumer speed on cold
> IO cache is like 5-50 times slower from what I can get with "sequential
> read" fio test.
>
> For kafka-consumer-perf-test I'm assuming that IO access pattern is
> sequential read.
>
> Here is FIO test:
>
> [sequential-read]
> rw=read
> size=50G
> ioengine=sync   # I know that kafka use sendfile, but sync should be
> slower, right?
> numjobs=1
>  directory=/tmp/fio
> filename=redo01.log
>
> Here what I'm doing with kafka-consumer-perf-test:
>
> kafka-consumer-perf-test.sh -topic "perf" --messages 50000000 --zookeeper
> host:2181 --threads 1 --socket-buffer-size 16777216 --fetch-size 16777216
>
> The broker config is  the same.
>
> I'm dropping IO cache before running tests: echo 3 >
> /proc/sys/vm/drop_caches
>
>
> -----------------------------------------------------------------------------------------------
> |           |   m1.xlarge            |    hi1.4xlarge              |
>  desktop    |
>
>  ---------------------------------------------------------------------------------------------
> |  kafka  |    25   MB/s           |     10  MB/s   (???)    |   20  MB/s
>  |
>
>  ---------------------------------------------------------------------------------------------
> |  fio      |   130   MB/s          |     450  MB/s             |    67