Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> application scenerio and suggested kafka setup


Copy link to this message
-
application scenerio and suggested kafka setup
Say I create web application/service where customers signup, and they place
some javascript on their website which will then send over http a message
to my servers every time someone clicks on a link on their website.

Each customer will send to their own custom subdomain like:

customer1.example.com/api/put?linkId=1&......

Say I have 100,000 customers.

1. If all events are of the same type, what are the potential means I could
partition my topics?  Or does it not make sense to?  I'm confused as to
what I am reading, is a given kafka topic + paritition combination ONLY
allowed to be consumed by a single consumer group?  If so, why is that?
 the kafka server can only handle a single thread connecting to it??

2. I will have a java servlet that will contain my producer (each front end
server will have the same servlet that will contain a producer).  I want to
batch every x messages.  From what I understand, my producer is something I
will create using a singleton correct?

3. I want my consumers to by dynamic in size, so during peak hours I want
to fire up more nodes to  keep up with traffic, is there a production
worthy consumer daemon that I can use (or learn from) that is open sourced
somewhere?

Much appreciated!

 
+
Jun Rao 2013-01-27, 23:54
+
Guy Doulberg 2013-01-28, 06:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB