Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka 0.7 performance compared to bare metal


Copy link to this message
-
Re: Kafka 0.7 performance compared to bare metal
It is hard to say where the bottleneck is just from your description. Would
it be possible for you to rerun the consumer test using hprof on the
consumer so we can understand whether the fetcher is waiting on the fetches
(i.e. the broker is the bottleneck) or on the enque (i.e. the consumer is
the bottleneck). Likewise for the producer test it would be good to do the
same for the broker and producer process to understand what is happening
there.

-Jay
On Fri, Aug 30, 2013 at 1:34 AM, Rafael Bagmanov <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am trying to understand how fast is kafka 0.7 compared to what I can get
> from hard drive. In essence I have 3 questions.
>
> In all tests below, I'm using single broker with single one-partitioned
> topic. Kafka perf tests have been run in 2 deployment configs:
> - broker, perf-test on same host
> - broker, perf-test on different hosts (the results are practically the
> same, so wont post them here)
>
>
> I'm using FIO(http://freecode.com/projects/fio) to benchmark speed of hard
> drives.
>
> Hardware I'm using:
> 1) m1.xlarge with ephemeral storage, 4 core cpu, 16 GB ram
> 2) hi1.4xlarge  with SSD, 16 core cpu, 64 GB ram
> 3) desktop machine with 7200 rpm sata, 4 core cpu, 8 GB ram
>
> Kafka broker config:
> Oracle jdk 1.6.0_38,  -Xmx2048
>
> socket.send.buffer=16777216
> socket.receive.buffer=16777216
> max.socket.request.bytes=104857600
> log.flush.interval=10000
> log.default.flush.interval.ms=1000
> log.default.flush.scheduler.interval.ms=1000
> num.threads=[num of cores]
>
>
> For kafka-producer-perf-test I'm assuming that IO access pattern is
> sequential write.
>
> Here is the test I ran with FIO:
>
> [sequential-write]
> rw=write
> size=50G
> ioengine=sync
> numjobs=1
> directory=/tmp/fio
> filename=redo01.log
>
>
> Here is kafka performance test:
>
> ./bin/kafka-producer-perf-test.sh -topic "perf" --batch-size 3000
> --messages 50000000 --message-size 1300 --brokerinfo
> broker.list=0:host:9092 --threads [number-of-cores]
>
>
> ----------------------------------------------------------------------------------------
> |           |   m1.xlarge            |    hi1.4xlarge       |  desktop
>  |
>
>  ----------------------------------------------------------------------------------------
> |  kafka  |     41 MB/s           |      217 MB/s       |     42 MB/s   |
>
>  -----------------------------------------------------------------------------------------
> |  fio      |     106 MB/s          |      377 MB/s       |    74 MB/s   |
>
> ----------------------------------------------------------------------------------------
>
>
> Question 1: The proportion (~1/2) is pretty stable against different kind
> of hardware I've tried. Is it as expected? Can something be done to improve
> this?
>
> I've tried to play with:
> log.flush.interval=10000
> log.default.flush.interval.ms=1000
> log.default.flush.scheduler.interval.ms=1000
>
> Like increasing 10 times, or decreasing 10 times, but haven't seen much of
> a difference in IO  throughput
>
> The other thing that bugs me much more is that kafka consumer speed on cold
> IO cache is like 5-50 times slower from what I can get with "sequential
> read" fio test.
>
> For kafka-consumer-perf-test I'm assuming that IO access pattern is
> sequential read.
>
> Here is FIO test:
>
> [sequential-read]
> rw=read
> size=50G
> ioengine=sync   # I know that kafka use sendfile, but sync should be
> slower, right?
> numjobs=1
>  directory=/tmp/fio
> filename=redo01.log
>
> Here what I'm doing with kafka-consumer-perf-test:
>
> kafka-consumer-perf-test.sh -topic "perf" --messages 50000000 --zookeeper
> host:2181 --threads 1 --socket-buffer-size 16777216 --fetch-size 16777216
>
> The broker config is  the same.
>
> I'm dropping IO cache before running tests: echo 3 >
> /proc/sys/vm/drop_caches
>
>
> -----------------------------------------------------------------------------------------------
> |           |   m1.xlarge            |    hi1.4xlarge              |