Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> kafka partitioning


Copy link to this message
-
Re: kafka partitioning
As I understand it, this partition is a broker_partion pair (in a multi
broker scenario).

again as I understand it : if , on the consumer side, stickness (same key
goes to same consumer)needs to be maintained  the same algorithm could be
used to figure out  which broker_partition to go against for fetch from a
consumer standpoint.

This works if the  brokers are not flapping (going up / down). If they are
flapping, then the consumer strategy would also need to change with the
flaps (s witch to different b_p pair).  On the "pull" side of the equation,
this becomes difficult because of tear_down/connect_up issues. is there a
better way?
On Sep 25, 2012 10:23 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:

> Producer uses a partitioning function to map partitionKey to a partition.
>
> Thanks,
>
> Jun
>
> On Tue, Sep 25, 2012 at 9:55 AM, Raymond Ng <[EMAIL PROTECTED]> wrote:
>
> > Hi
> >
> > can someone please explain the relationship between the partitioning key
> in
> > ProducerData(topic, *partitionKey*, eventList) from producer and the
> > partition parameter in FetchRequest(topic, *partition*, offset,
> fetchsize)
> > from consumer
> >
> > one is String and the other is int, so I assume there're not referring to
> > the same thing, I want to know if I can specify which partition on the
> > broker the producer's msg will be stored, and how to consume the  msg
> > specifically on this partition
> >
> > thanks
> >
> > Ray
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB