Kafka, mail # user - one consumerConnector or many? - 2013-05-29, 13:14
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
one consumerConnector or many?
In thinking about the design of consumption, we have in mind a generic
consumer server which would consume from more than one message type.  The
handling of each type of message would be different.  I suppose we could
have upwards of say 50 different message types, eventually, maybe 100+
different types.  Which of the following designs would be best and why would
the other options be bad?

 

1)      Have all message types go through one topic and use a dispatcher
pattern to select the correct handler.  Use one consumerConnector.

2)      Use a different topic for each message type, but still use one
consumerConnector and a dispatcher pattern.

3)      Use a different topic for each message type and have a separate
consumerConnector for each topic.

 

I am struggling with whether my assumptions are correct.  It seems that a
single connector for a topic would establish one socket to each broker, as
rebalancing assigns various partitions to that thread.  Option 2 would pull
messages from more than one topic through a single socket to a particular
broker, is it so?  Would option 3 be reasonable, establishing upwards of 100
sockets per broker?  

 

I am guestimating that option 2 is the right way forward, to bound socket
use, and we'll need to figure out a way to parameterize stream consumption
with the right handlers for a particular msg type.  If we add a topic, do
you think we should create a new connector or restart the original connector
with the new topic in the map?

 

Thanks,

rob
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB