I'm currently trying to understand how Kafka (0.8) can scale with our usage
pattern and how to setup the partitioning.
We want to route the same messages belonging to the same id to the same
queue, so its consumer will able to consume all the messages of that id.
- From my understanding, in Kafka we would need to have a custom
partitioner that routes the same messages to the same partition right? I'm
trying to find examples of writing this partitioner logic, but I can't find
any. Can someone point me to an example?
- I see that Kafka server.properties allows one to specify the number of
partitions it supports. However, when we want to scale I wonder if we add #
of partitions or # of brokers, will the same partitioner start distributing
the messages to different partitions?
And if it does, how can that same consumer continue to read off the
messages of those ids if it was interrupted in the middle?
- I'd like to create a consumer per partition, and for each one to
subscribe to the changes of that one. How can this be done in kafka?