Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Partitioning and scale


+
Timothy Chen 2013-05-22, 19:26
+
Chris Curtin 2013-05-22, 19:37
Copy link to this message
-
Re: Partitioning and scale
Neha Narkhede 2013-05-22, 20:15
- I see that Kafka server.properties allows one to specify the number of
partitions it supports. However, when we want to scale I wonder if we add #
of partitions or # of brokers, will the same partitioner start distributing
the messages to different partitions?
 And if it does, how can that same consumer continue to read off the
messages of those ids if it was interrupted in the middle?

The num.partitions config in server.properties is used only for topics that
are auto created (controlled by auto.create.topics.enable). For topics that
you create using the admin tool, you can specify the number of partitions
that you want. After that, currently there is no way to change that. For
that reason, it is a good idea to over partition your topic, which also
helps load balance partitions onto the brokers. You are right that if you
change the number of partitions later, then previously messages that stuck
to a certain partition would now get routed to a different partition, which
is undesirable for applications that want to use sticky partitioning.

- I'd like to create a consumer per partition, and for each one to
subscribe to the changes of that one. How can this be done in kafka?

For your use case, it seems like SimpleConsumer might be a better fit.
However, it will require you to write code to handle discovery of leader
for the partition that your consumer is consuming. Chris has written up a
great example that you can follow -
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example

Thanks,
Neha
On Wed, May 22, 2013 at 12:37 PM, Chris Curtin <[EMAIL PROTECTED]>wrote:

> Hi Tim,
>
>
> On Wed, May 22, 2013 at 3:25 PM, Timothy Chen <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I'm currently trying to understand how Kafka (0.8) can scale with our
> usage
> > pattern and how to setup the partitioning.
> >
> > We want to route the same messages belonging to the same id to the same
> > queue, so its consumer will able to consume all the messages of that id.
> >
> > My questions:
> >
> >  - From my understanding, in Kafka we would need to have a custom
> > partitioner that routes the same messages to the same partition right?
>  I'm
> > trying to find examples of writing this partitioner logic, but I can't
> find
> > any. Can someone point me to an example?
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example
>
> The partitioner here does a simple mod on the IP address and the # of
> partitions. You'd need to define your own logic, but this is a start.
>
>
> > - I see that Kafka server.properties allows one to specify the number of
> > partitions it supports. However, when we want to scale I wonder if we
> add #
> > of partitions or # of brokers, will the same partitioner start
> distributing
> > the messages to different partitions?
> >  And if it does, how can that same consumer continue to read off the
> > messages of those ids if it was interrupted in the middle?
> >
>
> I'll let someone else answer this.
>
>
> >
> > - I'd like to create a consumer per partition, and for each one to
> > subscribe to the changes of that one. How can this be done in kafka?
> >
>
> Two ways: Simple Consumer or Consumer Groups:
>
> Depends on the level of control you want on code processing a specific
> partition vs. getting one assigned to it (and level of control over offset
> management).
>
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> <https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example>
>
>
> >
> > Thanks,
> >
> > Tim
> >
>

 
+
Timothy Chen 2013-05-22, 21:20
+
Neha Narkhede 2013-05-22, 23:32
+
Timothy Chen 2013-05-23, 23:22
+
Milind Parikh 2013-05-23, 23:36
+
Neha Narkhede 2013-05-24, 15:40