Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Velocity on local machine


Copy link to this message
-
Re: Velocity on local machine
Klaus Schaefers 2014-01-13, 08:12
Hi,

I am using kafka-0.8.0-src.

Cheers,

Klaus
On Fri, Jan 10, 2014 at 8:48 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> What version of Kafka are you benchmarking?
>
>
> On Fri, Jan 10, 2014 at 8:36 AM, Klaus Schaefers <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > during my test cpu load is quite low, roughly 50 percent, sometimes peask
> > to 70%.
> >
> > >Are you using the sync producer per chance?
> > I enforced now the async and I got a huge improvement in one of my test
> > cases. let my explore the rest a little bit more.
> >
> > Cheers,
> >
> > Klaus
> >
> >
> >
> >
> > On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <[EMAIL PROTECTED]>
> > wrote:
> >
> > > 2k msgs/s is silly, unless your messages are 10MB each, so something is
> > > clearly wrong.
> > >
> > > * What is the CPU usage and IO load when running your performance
> tests?
> > > * Are you using the sync producer per chance? Maybe combined with an
> > > aggressive log.flush.interval?
> > > * For reference, and to find out where the bottleneck is, try running
> > > rdkafka_performance [1] according to the previous link I posted:
> > >   if performance does not increase then the broker is the problem,
> > > otherwise it is the producer/consumers.
> > >
> > > Regards,
> > > Magnus
> > >
> > > [1]: https://github.com/edenhill/librdkafka/tree/master/examples
> > >
> > >
> > >
> > > 2014/1/10 Klaus Schaefers <[EMAIL PROTECTED]>
> > >
> > > > Hi,
> > > >
> > > > I have close to 2k messages per second. My machine is just a (BG
> 4-core
> > > i5
> > > > but I would expect more messages. I ran Kafka in the default
> settings.
> > > >
> > > >
> > > > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > What performance numbers did you see?
> > > > >
> > > > > For reference you can check the following tests that were also run
> on
> > > the
> > > > > same machine as the broker:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > > > >
> > > > > Do they correspond to your numbers?
> > > > >
> > > > > Consumer thruput is not included in that document but typically
> peaks
> > > at
> > > > > around 3 million msgs/s when run on the same host and with hot disk
> > > > caches.
> > > > >
> > > > > Regards,
> > > > > Magnus
> > > > >
> > > > >
> > > > > 2014/1/10 Klaus Schaefers <[EMAIL PROTECTED]>
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am currently benchmarking Kafka against ActiveMQ and I got some
> > > > results
> > > > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > > > > messages
> > > > > > when running locally. But from all what I was reading I am a
> little
> > > bit
> > > > > > surprised. Honestly I expected Kafka to outperform ActiveMQ.
> Some I
> > > am
> > > > > left
> > > > > > with the feeling that I configured some wrong or did any kind of
> > > other
> > > > > > mistake.
> > > > > >
> > > > > > My setup looks like this:
> > > > > >
> > > > > > - One local broker
> > > > > > - 10 Topic / Queues
> > > > > > - 1 producer that dispatches messages randomly to the topics
> > > > > > - 1 consumer per topic
> > > > > >
> > > > > > I basically used the example on the Kafka web page.
> > > > > >
> > > > > > Also I encountered some issues when increasing the number of
> > topics,
> > > > lets
> > > > > > say 100. In this case the consumer cannot connect the
> Zookeeper...
> > > > > >
> > > > > > Does anybody has an idea how to improve the performance?
> > > > > >
> > > > > >
> > > > > > Thx,
> > > > > >
> > > > > > Klaus
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Klaus Schaefers
> > > > > > Senior Optimization Manager
> > > > > >
> > > > > > Ligatus GmbH
> > > > > > Hohenstaufenring 30-32
> > > > > > D-50674 Köln
> > > > > >
> > > > > > Tel.:  +49 (0) 221 / 56939 -784
> > > > > > Fax:  +49 (0) 221 / 56 939 - 599
> > > > > > E-Mail: [EMAIL PROTECTED]

Klaus Schaefers
Senior Optimization Manager

Ligatus GmbH
Hohenstaufenring 30-32
D-50674 Köln

Tel.:  +49 (0) 221 / 56939 -784
Fax:  +49 (0) 221 / 56 939 - 599
E-Mail: [EMAIL PROTECTED]
Web: www.ligatus.de

HRB Köln 56003
Geschäftsführung:
Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
Dipl.-Wirtschaftsingenieur Arne Wolter