Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Velocity on local machine


+
Klaus Schaefers 2014-01-10, 09:44
+
Magnus Edenhill 2014-01-10, 11:33
+
Klaus Schaefers 2014-01-10, 12:07
+
Gerrit Jansen van Vuuren 2014-01-10, 13:27
Copy link to this message
-
Re: Velocity on local machine
2k msgs/s is silly, unless your messages are 10MB each, so something is
clearly wrong.

* What is the CPU usage and IO load when running your performance tests?
* Are you using the sync producer per chance? Maybe combined with an
aggressive log.flush.interval?
* For reference, and to find out where the bottleneck is, try running
rdkafka_performance [1] according to the previous link I posted:
  if performance does not increase then the broker is the problem,
otherwise it is the producer/consumers.

Regards,
Magnus

[1]: https://github.com/edenhill/librdkafka/tree/master/examples

2014/1/10 Klaus Schaefers <[EMAIL PROTECTED]>

> Hi,
>
> I have close to 2k messages per second. My machine is just a (BG 4-core i5
> but I would expect more messages. I ran Kafka in the default settings.
>
>
> On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <[EMAIL PROTECTED]
> >wrote:
>
> > What performance numbers did you see?
> >
> > For reference you can check the following tests that were also run on the
> > same machine as the broker:
> >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> >
> > Do they correspond to your numbers?
> >
> > Consumer thruput is not included in that document but typically peaks at
> > around 3 million msgs/s when run on the same host and with hot disk
> caches.
> >
> > Regards,
> > Magnus
> >
> >
> > 2014/1/10 Klaus Schaefers <[EMAIL PROTECTED]>
> >
> > > Hi,
> > >
> > > I am currently benchmarking Kafka against ActiveMQ and I got some
> results
> > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > messages
> > > when running locally. But from all what I was reading I am a little bit
> > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I am
> > left
> > > with the feeling that I configured some wrong or did any kind of other
> > > mistake.
> > >
> > > My setup looks like this:
> > >
> > > - One local broker
> > > - 10 Topic / Queues
> > > - 1 producer that dispatches messages randomly to the topics
> > > - 1 consumer per topic
> > >
> > > I basically used the example on the Kafka web page.
> > >
> > > Also I encountered some issues when increasing the number of topics,
> lets
> > > say 100. In this case the consumer cannot connect the Zookeeper...
> > >
> > > Does anybody has an idea how to improve the performance?
> > >
> > >
> > > Thx,
> > >
> > > Klaus
> > >
> > >
> > > --
> > >
> > > --
> > >
> > > Klaus Schaefers
> > > Senior Optimization Manager
> > >
> > > Ligatus GmbH
> > > Hohenstaufenring 30-32
> > > D-50674 Köln
> > >
> > > Tel.:  +49 (0) 221 / 56939 -784
> > > Fax:  +49 (0) 221 / 56 939 - 599
> > > E-Mail: [EMAIL PROTECTED]
> > > Web: www.ligatus.de
> > >
> > > HRB Köln 56003
> > > Geschäftsführung:
> > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > >
> >
>
>
>
> --
>
> --
>
> Klaus Schaefers
> Senior Optimization Manager
>
> Ligatus GmbH
> Hohenstaufenring 30-32
> D-50674 Köln
>
> Tel.:  +49 (0) 221 / 56939 -784
> Fax:  +49 (0) 221 / 56 939 - 599
> E-Mail: [EMAIL PROTECTED]
> Web: www.ligatus.de
>
> HRB Köln 56003
> Geschäftsführung:
> Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> Dipl.-Wirtschaftsingenieur Arne Wolter
>

 
+
Klaus Schaefers 2014-01-10, 16:37
+
Neha Narkhede 2014-01-10, 19:48
+
Klaus Schaefers 2014-01-13, 08:12
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB