Markus Roder 2013-10-08, 09:12
Neha Narkhede 2013-10-08, 14:28
thanks for this information.
Can you give me a hint for implementing a own rebalancing strategy?
Thanks in advance
2013/10/8 Neha Narkhede <[EMAIL PROTECTED]>
> Currently there is no way to invoke a callback on the rebalance operation.
> But this is certainly something to consider for Kafka 0.9 since we are
> planning a client rewrite for that release. You can find the proposal in
> progress here -
> For now your best bet is to use the SimpleConsumer and implement your own
> rebalancing strategy. Another hacky approach is to register zookeeper
> watches on the /consumers/<group>/owners path that indicates the partition
> ownership change.
> On Oct 8, 2013 2:12 AM, "Markus Roder" <[EMAIL PROTECTED]> wrote:
> > Hi,
> > we currently face a "problem" on our consumer cluster, which may have a
> > simple solution. Never the less I could not find this solution yet.
> > Description of problem:
> > 1 kafka topic with 24 partitions (kafka version 0.8 Beta1
> > 2 or more consumers in same consumer group. Each consumer processes its
> > partitions by aggregating topic data into a relational database. Each
> > consumer hashes the aggregation data locally for commiting data into the
> > relational database. After commit to database the consumerConnector
> > the offsets to kafka.
> > Problem is: If I connect a new consumer, the consumerconnector
> > the partitions to read from on each consumer instance. That causes our
> > system currently to process topic-data twice because of the local
> > aggregation within the consumer.
> > Is there any possibility to catch the event of new partition selection in
> > conumserConnector to commit the offsets and data before reconnecting to
> > partitions?
> > Thanks in advance
> > Markus
> > --
> > Markus Roder
> > Distelweg 4
> > 97318 Kitzingen
> > Mail: [EMAIL PROTECTED]
> > Profil: http://gplus.to/markusroder
Mail: [EMAIL PROTECTED]
Neha Narkhede 2013-10-09, 16:32
Markus Roder 2013-10-11, 10:00