Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Velocity on local machine


Copy link to this message
-
Re: Velocity on local machine
Hi,

during my test cpu load is quite low, roughly 50 percent, sometimes peask
to 70%.

>Are you using the sync producer per chance?
I enforced now the async and I got a huge improvement in one of my test
cases. let my explore the rest a little bit more.

Cheers,

Klaus
On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <[EMAIL PROTECTED]> wrote:

> 2k msgs/s is silly, unless your messages are 10MB each, so something is
> clearly wrong.
>
> * What is the CPU usage and IO load when running your performance tests?
> * Are you using the sync producer per chance? Maybe combined with an
> aggressive log.flush.interval?
> * For reference, and to find out where the bottleneck is, try running
> rdkafka_performance [1] according to the previous link I posted:
>   if performance does not increase then the broker is the problem,
> otherwise it is the producer/consumers.
>
> Regards,
> Magnus
>
> [1]: https://github.com/edenhill/librdkafka/tree/master/examples
>
>
>
> 2014/1/10 Klaus Schaefers <[EMAIL PROTECTED]>
>
> > Hi,
> >
> > I have close to 2k messages per second. My machine is just a (BG 4-core
> i5
> > but I would expect more messages. I ran Kafka in the default settings.
> >
> >
> > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <[EMAIL PROTECTED]
> > >wrote:
> >
> > > What performance numbers did you see?
> > >
> > > For reference you can check the following tests that were also run on
> the
> > > same machine as the broker:
> > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > >
> > > Do they correspond to your numbers?
> > >
> > > Consumer thruput is not included in that document but typically peaks
> at
> > > around 3 million msgs/s when run on the same host and with hot disk
> > caches.
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > > 2014/1/10 Klaus Schaefers <[EMAIL PROTECTED]>
> > >
> > > > Hi,
> > > >
> > > > I am currently benchmarking Kafka against ActiveMQ and I got some
> > results
> > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > > messages
> > > > when running locally. But from all what I was reading I am a little
> bit
> > > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I
> am
> > > left
> > > > with the feeling that I configured some wrong or did any kind of
> other
> > > > mistake.
> > > >
> > > > My setup looks like this:
> > > >
> > > > - One local broker
> > > > - 10 Topic / Queues
> > > > - 1 producer that dispatches messages randomly to the topics
> > > > - 1 consumer per topic
> > > >
> > > > I basically used the example on the Kafka web page.
> > > >
> > > > Also I encountered some issues when increasing the number of topics,
> > lets
> > > > say 100. In this case the consumer cannot connect the Zookeeper...
> > > >
> > > > Does anybody has an idea how to improve the performance?
> > > >
> > > >
> > > > Thx,
> > > >
> > > > Klaus
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > >
> > > > Klaus Schaefers
> > > > Senior Optimization Manager
> > > >
> > > > Ligatus GmbH
> > > > Hohenstaufenring 30-32
> > > > D-50674 Köln
> > > >
> > > > Tel.:  +49 (0) 221 / 56939 -784
> > > > Fax:  +49 (0) 221 / 56 939 - 599
> > > > E-Mail: [EMAIL PROTECTED]
> > > > Web: www.ligatus.de
> > > >
> > > > HRB Köln 56003
> > > > Geschäftsführung:
> > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > > >
> > >
> >
> >
> >
> > --
> >
> > --
> >
> > Klaus Schaefers
> > Senior Optimization Manager
> >
> > Ligatus GmbH
> > Hohenstaufenring 30-32
> > D-50674 Köln
> >
> > Tel.:  +49 (0) 221 / 56939 -784
> > Fax:  +49 (0) 221 / 56 939 - 599
> > E-Mail: [EMAIL PROTECTED]
> > Web: www.ligatus.de
> >
> > HRB Köln 56003
> > Geschäftsführung:
> > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > Dipl.-Wirtschaftsingenieur Arne Wolter
> >
>

--

--

Klaus Schaefers
Senior Optimization Manager

Ligatus GmbH
Hohenstaufenring 30-32
D-50674 Köln

Tel.:  +49 (0) 221 / 56939 -784
Fax:  +49 (0) 221 / 56 939 - 599
E-Mail: [EMAIL PROTECTED]
Web: www.ligatus.de

HRB Köln 56003
Geschäftsführung:
Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
Dipl.-Wirtschaftsingenieur Arne Wolter

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB