Kafka, mail # user - application scenerio and suggested kafka setup - 2013-01-27, 21:07
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
Copy link to this message
application scenerio and suggested kafka setup
Say I create web application/service where customers signup, and they place
some javascript on their website which will then send over http a message
to my servers every time someone clicks on a link on their website.

Each customer will send to their own custom subdomain like:


Say I have 100,000 customers.

1. If all events are of the same type, what are the potential means I could
partition my topics?  Or does it not make sense to?  I'm confused as to
what I am reading, is a given kafka topic + paritition combination ONLY
allowed to be consumed by a single consumer group?  If so, why is that?
 the kafka server can only handle a single thread connecting to it??

2. I will have a java servlet that will contain my producer (each front end
server will have the same servlet that will contain a producer).  I want to
batch every x messages.  From what I understand, my producer is something I
will create using a singleton correct?

3. I want my consumers to by dynamic in size, so during peak hours I want
to fire up more nodes to  keep up with traffic, is there a production
worthy consumer daemon that I can use (or learn from) that is open sourced

Much appreciated!

Jun Rao 2013-01-27, 23:54
Guy Doulberg 2013-01-28, 06:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB