Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Partitioning and scale


Copy link to this message
-
Partitioning and scale
Timothy Chen 2013-05-22, 19:26
Hi,

I'm currently trying to understand how Kafka (0.8) can scale with our usage
pattern and how to setup the partitioning.

We want to route the same messages belonging to the same id to the same
queue, so its consumer will able to consume all the messages of that id.

My questions:

 - From my understanding, in Kafka we would need to have a custom
partitioner that routes the same messages to the same partition right?  I'm
trying to find examples of writing this partitioner logic, but I can't find
any. Can someone point me to an example?

- I see that Kafka server.properties allows one to specify the number of
partitions it supports. However, when we want to scale I wonder if we add #
of partitions or # of brokers, will the same partitioner start distributing
the messages to different partitions?
 And if it does, how can that same consumer continue to read off the
messages of those ids if it was interrupted in the middle?

- I'd like to create a consumer per partition, and for each one to
subscribe to the changes of that one. How can this be done in kafka?

Thanks,

Tim

 
+
Chris Curtin 2013-05-22, 19:37
+
Neha Narkhede 2013-05-22, 20:15
+
Timothy Chen 2013-05-22, 21:20
+
Neha Narkhede 2013-05-22, 23:32
+
Timothy Chen 2013-05-23, 23:22
+
Milind Parikh 2013-05-23, 23:36
+
Neha Narkhede 2013-05-24, 15:40