Home | About | Sematext search-lucene.com search-hadoop.com search-devops.com metrics + logs = try SPM and Logsene for free
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> one consumerConnector or many?


Copy link to this message
-
Re: one consumerConnector or many?
I'd look at a variation of #2. Can your messages by grouped into a 'class
(for lack of a better term)' that are consumed together? For example a
'class' of 'auditing events' or 'sensor events'. The idea would to then
have a topic for 'class'.

A couple of benefits to this:
- you can define your consumption of a 'class's resources by value. So the
'audit' topic may only get a 2 threaded consumer while the 'sensor' class
gets a 10 threaded consumer.
- you can stop processing a 'class' of messages if you need to without
taking all the consumers off line (Assuming you have different processors
or a way while running to alter your number of threads per topic.)

Since it sounds like you may be frequently adding new message types this
approach also allows you to decide if you want to shutdown only a part of
your processing to add the new code to handle the message.

Finally, why the concern about socket use? A well configured Windows or
Linux machine can have thousands of open sockets without problems. Since
0.8.0 only connects to the Broker with the topic/partition you end up with
1 socket per topic/partition and consumer.

Hope this helps,

Chris
On Wed, May 29, 2013 at 9:13 AM, Rob Withers <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB