Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Partition election on consumer

Copy link to this message
Re: Partition election on consumer
Currently there is no way to invoke a callback on the rebalance operation.
But this is certainly something to consider for Kafka 0.9 since we are
planning a client rewrite for that release. You can find the proposal in
progress here -

For now your best bet is to use the SimpleConsumer and implement your own
rebalancing strategy. Another hacky approach is to register zookeeper
watches on the /consumers/<group>/owners path that indicates the partition
ownership change.

On Oct 8, 2013 2:12 AM, "Markus Roder" <[EMAIL PROTECTED]> wrote:

> Hi,
> we currently face a "problem" on our consumer cluster, which may have a
> simple solution. Never the less I could not find this solution yet.
> Description of problem:
> 1 kafka topic with 24 partitions (kafka version 0.8 Beta1
> 2 or more consumers in same consumer group. Each consumer processes its
> partitions by aggregating topic data into a relational database. Each
> consumer hashes the aggregation data locally for commiting data into the
> relational database. After commit to database the consumerConnector commits
> the offsets to kafka.
> Problem is: If I connect a new consumer, the consumerconnector recalculates
> the partitions to read from on each consumer instance. That causes our
> system currently to process topic-data twice because of the local
> aggregation within the consumer.
> Is there any possibility to catch the event of new partition selection in
> conumserConnector to commit the offsets and data before reconnecting to new
> partitions?
> Thanks in advance
> Markus
> --
> Markus Roder
> Distelweg 4
> 97318 Kitzingen
> Profil: http://gplus.to/markusroder