Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Partition election on consumer


Copy link to this message
-
Re: Partition election on consumer
Hi Neha,

thanks for this information.
Can you give me a hint for implementing a own rebalancing strategy?

Thanks in advance
Markus
2013/10/8 Neha Narkhede <[EMAIL PROTECTED]>

> Currently there is no way to invoke a callback on the rebalance operation.
> But this is certainly something to consider for Kafka 0.9 since we are
> planning a client rewrite for that release. You can find the proposal in
> progress here -
>
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
>
> For now your best bet is to use the SimpleConsumer and implement your own
> rebalancing strategy. Another hacky approach is to register zookeeper
> watches on the /consumers/<group>/owners path that indicates the partition
> ownership change.
>
> Thanks,
> Neha
> On Oct 8, 2013 2:12 AM, "Markus Roder" <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > we currently face a "problem" on our consumer cluster, which may have a
> > simple solution. Never the less I could not find this solution yet.
> >
> > Description of problem:
> > 1 kafka topic with 24 partitions (kafka version 0.8 Beta1
> > 2 or more consumers in same consumer group. Each consumer processes its
> > partitions by aggregating topic data into a relational database. Each
> > consumer hashes the aggregation data locally for commiting data into the
> > relational database. After commit to database the consumerConnector
> commits
> > the offsets to kafka.
> >
> > Problem is: If I connect a new consumer, the consumerconnector
> recalculates
> > the partitions to read from on each consumer instance. That causes our
> > system currently to process topic-data twice because of the local
> > aggregation within the consumer.
> >
> > Is there any possibility to catch the event of new partition selection in
> > conumserConnector to commit the offsets and data before reconnecting to
> new
> > partitions?
> >
> > Thanks in advance
> > Markus
> >
> > --
> > Markus Roder
> > Distelweg 4
> > 97318 Kitzingen
> > Mail: [EMAIL PROTECTED]
> > Profil: http://gplus.to/markusroder
> >
>

--
Markus Roder
Distelweg 4
97318 Kitzingen
Mail: [EMAIL PROTECTED]
Profil: http://gplus.to/markusroder