Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Partition election on consumer


+
Markus Roder 2013-10-08, 09:12
+
Neha Narkhede 2013-10-08, 14:28
Copy link to this message
-
Re: Partition election on consumer
Hi Neha,

thanks for this information.
Can you give me a hint for implementing a own rebalancing strategy?

Thanks in advance
Markus
2013/10/8 Neha Narkhede <[EMAIL PROTECTED]>

> Currently there is no way to invoke a callback on the rebalance operation.
> But this is certainly something to consider for Kafka 0.9 since we are
> planning a client rewrite for that release. You can find the proposal in
> progress here -
>
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
>
> For now your best bet is to use the SimpleConsumer and implement your own
> rebalancing strategy. Another hacky approach is to register zookeeper
> watches on the /consumers/<group>/owners path that indicates the partition
> ownership change.
>
> Thanks,
> Neha
> On Oct 8, 2013 2:12 AM, "Markus Roder" <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > we currently face a "problem" on our consumer cluster, which may have a
> > simple solution. Never the less I could not find this solution yet.
> >
> > Description of problem:
> > 1 kafka topic with 24 partitions (kafka version 0.8 Beta1
> > 2 or more consumers in same consumer group. Each consumer processes its
> > partitions by aggregating topic data into a relational database. Each
> > consumer hashes the aggregation data locally for commiting data into the
> > relational database. After commit to database the consumerConnector
> commits
> > the offsets to kafka.
> >
> > Problem is: If I connect a new consumer, the consumerconnector
> recalculates
> > the partitions to read from on each consumer instance. That causes our
> > system currently to process topic-data twice because of the local
> > aggregation within the consumer.
> >
> > Is there any possibility to catch the event of new partition selection in
> > conumserConnector to commit the offsets and data before reconnecting to
> new
> > partitions?
> >
> > Thanks in advance
> > Markus
> >
> > --
> > Markus Roder
> > Distelweg 4
> > 97318 Kitzingen
> > Mail: [EMAIL PROTECTED]
> > Profil: http://gplus.to/markusroder
> >
>

--
Markus Roder
Distelweg 4
97318 Kitzingen
Mail: [EMAIL PROTECTED]
Profil: http://gplus.to/markusroder

 
+
Neha Narkhede 2013-10-09, 16:32
+
Markus Roder 2013-10-11, 10:00
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB