Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> kafka partitioning

Copy link to this message
Re: kafka partitioning
As I understand it, this partition is a broker_partion pair (in a multi
broker scenario).

again as I understand it : if , on the consumer side, stickness (same key
goes to same consumer)needs to be maintained  the same algorithm could be
used to figure out  which broker_partition to go against for fetch from a
consumer standpoint.

This works if the  brokers are not flapping (going up / down). If they are
flapping, then the consumer strategy would also need to change with the
flaps (s witch to different b_p pair).  On the "pull" side of the equation,
this becomes difficult because of tear_down/connect_up issues. is there a
better way?
On Sep 25, 2012 10:23 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:

> Producer uses a partitioning function to map partitionKey to a partition.
> Thanks,
> Jun
> On Tue, Sep 25, 2012 at 9:55 AM, Raymond Ng <[EMAIL PROTECTED]> wrote:
> > Hi
> >
> > can someone please explain the relationship between the partitioning key
> in
> > ProducerData(topic, *partitionKey*, eventList) from producer and the
> > partition parameter in FetchRequest(topic, *partition*, offset,
> fetchsize)
> > from consumer
> >
> > one is String and the other is int, so I assume there're not referring to
> > the same thing, I want to know if I can specify which partition on the
> > broker the producer's msg will be stored, and how to consume the  msg
> > specifically on this partition
> >
> > thanks
> >
> > Ray
> >