Kafka, mail # user - Partitions and highlevel consumers - 2013-07-02, 18:55
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
Copy link to this message
Partitions and highlevel consumers
Hi guys, we are starting with kafka in our project. We are using version
0.8. I come from a traditional JMS MoM architecture, and some things are
new to me.

One thing that I'm not getting is the mapping between partitions and number
of threads. On a single consumer I can see the relationship but what
happens when you have multiple consumers?

So say, I have a topic and I have 5 instances of services that acts as
consumers. On a traditional pub/sub I would have all 5 instances to consume
the message right? Even if this topic only has one partition and now we
have 5 threads (one per jvm instance) consuming it. Does this proceed?

Just want to make sure if that is the expected behavior, from the docs it
wasn't clear if its the number of threads per consumer, or the total number
of threads around all clients to the topic.


The intuitive mind is a sacred gift and the
rational mind is a faithful servant. We have
created a society that honors the servant and
has forgotten the gift.

Josh Foure 2013-07-02, 19:35
Jay Kreps 2013-07-02, 19:52
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB