Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Velocity on local machine


Copy link to this message
-
Re: Velocity on local machine
Klaus Schaefers 2014-01-10, 16:37
Hi,

during my test cpu load is quite low, roughly 50 percent, sometimes peask
to 70%.

>Are you using the sync producer per chance?
I enforced now the async and I got a huge improvement in one of my test
cases. let my explore the rest a little bit more.

Cheers,

Klaus
On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <[EMAIL PROTECTED]> wrote:

> 2k msgs/s is silly, unless your messages are 10MB each, so something is
> clearly wrong.
>
> * What is the CPU usage and IO load when running your performance tests?
> * Are you using the sync producer per chance? Maybe combined with an
> aggressive log.flush.interval?
> * For reference, and to find out where the bottleneck is, try running
> rdkafka_performance [1] according to the previous link I posted:
>   if performance does not increase then the broker is the problem,
> otherwise it is the producer/consumers.
>
> Regards,
> Magnus
>
> [1]: https://github.com/edenhill/librdkafka/tree/master/examples
>
>
>
> 2014/1/10 Klaus Schaefers <[EMAIL PROTECTED]>
>
> > Hi,
> >
> > I have close to 2k messages per second. My machine is just a (BG 4-core
> i5
> > but I would expect more messages. I ran Kafka in the default settings.
> >
> >
> > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <[EMAIL PROTECTED]
> > >wrote:
> >
> > > What performance numbers did you see?
> > >
> > > For reference you can check the following tests that were also run on
> the
> > > same machine as the broker:
> > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > >
> > > Do they correspond to your numbers?
> > >
> > > Consumer thruput is not included in that document but typically peaks
> at
> > > around 3 million msgs/s when run on the same host and with hot disk
> > caches.
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > > 2014/1/10 Klaus Schaefers <[EMAIL PROTECTED]>
> > >
> > > > Hi,
> > > >
> > > > I am currently benchmarking Kafka against ActiveMQ and I got some
> > results
> > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > > messages
> > > > when running locally. But from all what I was reading I am a little
> bit
> > > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I
> am
> > > left
> > > > with the feeling that I configured some wrong or did any kind of
> other
> > > > mistake.
> > > >
> > > > My setup looks like this:
> > > >
> > > > - One local broker
> > > > - 10 Topic / Queues
> > > > - 1 producer that dispatches messages randomly to the topics
> > > > - 1 consumer per topic
> > > >
> > > > I basically used the example on the Kafka web page.
> > > >
> > > > Also I encountered some issues when increasing the number of topics,
> > lets
> > > > say 100. In this case the consumer cannot connect the Zookeeper...
> > > >
> > > > Does anybody has an idea how to improve the performance?
> > > >
> > > >
> > > > Thx,
> > > >
> > > > Klaus
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > >
> > > > Klaus Schaefers
> > > > Senior Optimization Manager
> > > >
> > > > Ligatus GmbH
> > > > Hohenstaufenring 30-32
> > > > D-50674 Köln
> > > >
> > > > Tel.:  +49 (0) 221 / 56939 -784
> > > > Fax:  +49 (0) 221 / 56 939 - 599
> > > > E-Mail: [EMAIL PROTECTED]
> > > > Web: www.ligatus.de
> > > >
> > > > HRB Köln 56003
> > > > Geschäftsführung:
> > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > > >
> > >
> >
> >
> >
> > --
> >
> > --
> >
> > Klaus Schaefers
> > Senior Optimization Manager
> >
> > Ligatus GmbH
> > Hohenstaufenring 30-32
> > D-50674 Köln
> >
> > Tel.:  +49 (0) 221 / 56939 -784
> > Fax:  +49 (0) 221 / 56 939 - 599
> > E-Mail: [EMAIL PROTECTED]
> > Web: www.ligatus.de
> >
> > HRB Köln 56003
> > Geschäftsführung:
> > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > Dipl.-Wirtschaftsingenieur Arne Wolter
> >
>

--

--

Klaus Schaefers
Senior Optimization Manager

Ligatus GmbH
Hohenstaufenring 30-32
D-50674 Köln

Tel.:  +49 (0) 221 / 56939 -784
Fax:  +49 (0) 221 / 56 939 - 599
E-Mail: [EMAIL PROTECTED]
Web: www.ligatus.de

HRB Köln 56003
Geschäftsführung:
Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
Dipl.-Wirtschaftsingenieur Arne Wolter