Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Partition election on consumer


+
Markus Roder 2013-10-08, 09:12
Copy link to this message
-
Re: Partition election on consumer
Currently there is no way to invoke a callback on the rebalance operation.
But this is certainly something to consider for Kafka 0.9 since we are
planning a client rewrite for that release. You can find the proposal in
progress here -
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI

For now your best bet is to use the SimpleConsumer and implement your own
rebalancing strategy. Another hacky approach is to register zookeeper
watches on the /consumers/<group>/owners path that indicates the partition
ownership change.

Thanks,
Neha
On Oct 8, 2013 2:12 AM, "Markus Roder" <[EMAIL PROTECTED]> wrote:

> Hi,
>
> we currently face a "problem" on our consumer cluster, which may have a
> simple solution. Never the less I could not find this solution yet.
>
> Description of problem:
> 1 kafka topic with 24 partitions (kafka version 0.8 Beta1
> 2 or more consumers in same consumer group. Each consumer processes its
> partitions by aggregating topic data into a relational database. Each
> consumer hashes the aggregation data locally for commiting data into the
> relational database. After commit to database the consumerConnector commits
> the offsets to kafka.
>
> Problem is: If I connect a new consumer, the consumerconnector recalculates
> the partitions to read from on each consumer instance. That causes our
> system currently to process topic-data twice because of the local
> aggregation within the consumer.
>
> Is there any possibility to catch the event of new partition selection in
> conumserConnector to commit the offsets and data before reconnecting to new
> partitions?
>
> Thanks in advance
> Markus
>
> --
> Markus Roder
> Distelweg 4
> 97318 Kitzingen
> Mail: [EMAIL PROTECTED]
> Profil: http://gplus.to/markusroder
>

 
+
Markus Roder 2013-10-09, 06:42
+
Neha Narkhede 2013-10-09, 16:32
+
Markus Roder 2013-10-11, 10:00
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB