Kafka, mail # user - Partitioning and scale - 2013-05-22, 19:26
 Search Hadoop and all its subprojects:

Switch to Plain View
Copy link to this message
Partitioning and scale

I'm currently trying to understand how Kafka (0.8) can scale with our usage
pattern and how to setup the partitioning.

We want to route the same messages belonging to the same id to the same
queue, so its consumer will able to consume all the messages of that id.

My questions:

 - From my understanding, in Kafka we would need to have a custom
partitioner that routes the same messages to the same partition right?  I'm
trying to find examples of writing this partitioner logic, but I can't find
any. Can someone point me to an example?

- I see that Kafka server.properties allows one to specify the number of
partitions it supports. However, when we want to scale I wonder if we add #
of partitions or # of brokers, will the same partitioner start distributing
the messages to different partitions?
 And if it does, how can that same consumer continue to read off the
messages of those ids if it was interrupted in the middle?

- I'd like to create a consumer per partition, and for each one to
subscribe to the changes of that one. How can this be done in kafka?



Chris Curtin 2013-05-22, 19:37
Neha Narkhede 2013-05-22, 20:15
Timothy Chen 2013-05-22, 21:20
Neha Narkhede 2013-05-22, 23:32
Timothy Chen 2013-05-23, 23:22
Milind Parikh 2013-05-23, 23:36
Neha Narkhede 2013-05-24, 15:40
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB