Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> application scenerio and suggested kafka setup


+
S Ahmed 2013-01-27, 21:07
Copy link to this message
-
Re: application scenerio and suggested kafka setup
Partition is useful for increasing the degree of parallelism of consumers
and to a certain degree, producers too. You can have multiple consumer
groups consuming the same topic/partition.

Thanks,

Jun

On Sun, Jan 27, 2013 at 1:06 PM, S Ahmed <[EMAIL PROTECTED]> wrote:

> Say I create web application/service where customers signup, and they place
> some javascript on their website which will then send over http a message
> to my servers every time someone clicks on a link on their website.
>
> Each customer will send to their own custom subdomain like:
>
> customer1.example.com/api/put?linkId=1&......
>
> Say I have 100,000 customers.
>
> 1. If all events are of the same type, what are the potential means I could
> partition my topics?  Or does it not make sense to?  I'm confused as to
> what I am reading, is a given kafka topic + paritition combination ONLY
> allowed to be consumed by a single consumer group?  If so, why is that?
>  the kafka server can only handle a single thread connecting to it??
>
> 2. I will have a java servlet that will contain my producer (each front end
> server will have the same servlet that will contain a producer).  I want to
> batch every x messages.  From what I understand, my producer is something I
> will create using a singleton correct?
>
> 3. I want my consumers to by dynamic in size, so during peak hours I want
> to fire up more nodes to  keep up with traffic, is there a production
> worthy consumer daemon that I can use (or learn from) that is open sourced
> somewhere?
>
> Much appreciated!
>

 
+
Guy Doulberg 2013-01-28, 06:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB