Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Partition election on consumer


Copy link to this message
-
Re: Partition election on consumer
Kafka's consumer rebalancing strategy is explained in detail here -
http://kafka.apache.org/documentation.html#distributionimpl
Hope that helps!

-Neha
On Tue, Oct 8, 2013 at 11:42 PM, Markus Roder <[EMAIL PROTECTED]>wrote:

> Hi Neha,
>
> thanks for this information.
> Can you give me a hint for implementing a own rebalancing strategy?
>
> Thanks in advance
> Markus
>
>
> 2013/10/8 Neha Narkhede <[EMAIL PROTECTED]>
>
> > Currently there is no way to invoke a callback on the rebalance
> operation.
> > But this is certainly something to consider for Kafka 0.9 since we are
> > planning a client rewrite for that release. You can find the proposal in
> > progress here -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
> >
> > For now your best bet is to use the SimpleConsumer and implement your own
> > rebalancing strategy. Another hacky approach is to register zookeeper
> > watches on the /consumers/<group>/owners path that indicates the
> partition
> > ownership change.
> >
> > Thanks,
> > Neha
> > On Oct 8, 2013 2:12 AM, "Markus Roder" <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > we currently face a "problem" on our consumer cluster, which may have a
> > > simple solution. Never the less I could not find this solution yet.
> > >
> > > Description of problem:
> > > 1 kafka topic with 24 partitions (kafka version 0.8 Beta1
> > > 2 or more consumers in same consumer group. Each consumer processes its
> > > partitions by aggregating topic data into a relational database. Each
> > > consumer hashes the aggregation data locally for commiting data into
> the
> > > relational database. After commit to database the consumerConnector
> > commits
> > > the offsets to kafka.
> > >
> > > Problem is: If I connect a new consumer, the consumerconnector
> > recalculates
> > > the partitions to read from on each consumer instance. That causes our
> > > system currently to process topic-data twice because of the local
> > > aggregation within the consumer.
> > >
> > > Is there any possibility to catch the event of new partition selection
> in
> > > conumserConnector to commit the offsets and data before reconnecting to
> > new
> > > partitions?
> > >
> > > Thanks in advance
> > > Markus
> > >
> > > --
> > > Markus Roder
> > > Distelweg 4
> > > 97318 Kitzingen
> > > Mail: [EMAIL PROTECTED]
> > > Profil: http://gplus.to/markusroder
> > >
> >
>
>
>
> --
> Markus Roder
> Distelweg 4
> 97318 Kitzingen
> Mail: [EMAIL PROTECTED]
> Profil: http://gplus.to/markusroder
>