Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Partition election on consumer


+
Markus Roder 2013-10-08, 09:12
+
Neha Narkhede 2013-10-08, 14:28
+
Markus Roder 2013-10-09, 06:42
+
Neha Narkhede 2013-10-09, 16:32
Copy link to this message
-
Re: Partition election on consumer
Thanks Neha, really appreciate your assistance
2013/10/9 Neha Narkhede <[EMAIL PROTECTED]>

> Kafka's consumer rebalancing strategy is explained in detail here -
> http://kafka.apache.org/documentation.html#distributionimpl
> Hope that helps!
>
> -Neha
>
>
> On Tue, Oct 8, 2013 at 11:42 PM, Markus Roder <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Neha,
> >
> > thanks for this information.
> > Can you give me a hint for implementing a own rebalancing strategy?
> >
> > Thanks in advance
> > Markus
> >
> >
> > 2013/10/8 Neha Narkhede <[EMAIL PROTECTED]>
> >
> > > Currently there is no way to invoke a callback on the rebalance
> > operation.
> > > But this is certainly something to consider for Kafka 0.9 since we are
> > > planning a client rewrite for that release. You can find the proposal
> in
> > > progress here -
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
> > >
> > > For now your best bet is to use the SimpleConsumer and implement your
> own
> > > rebalancing strategy. Another hacky approach is to register zookeeper
> > > watches on the /consumers/<group>/owners path that indicates the
> > partition
> > > ownership change.
> > >
> > > Thanks,
> > > Neha
> > > On Oct 8, 2013 2:12 AM, "Markus Roder" <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > we currently face a "problem" on our consumer cluster, which may
> have a
> > > > simple solution. Never the less I could not find this solution yet.
> > > >
> > > > Description of problem:
> > > > 1 kafka topic with 24 partitions (kafka version 0.8 Beta1
> > > > 2 or more consumers in same consumer group. Each consumer processes
> its
> > > > partitions by aggregating topic data into a relational database. Each
> > > > consumer hashes the aggregation data locally for commiting data into
> > the
> > > > relational database. After commit to database the consumerConnector
> > > commits
> > > > the offsets to kafka.
> > > >
> > > > Problem is: If I connect a new consumer, the consumerconnector
> > > recalculates
> > > > the partitions to read from on each consumer instance. That causes
> our
> > > > system currently to process topic-data twice because of the local
> > > > aggregation within the consumer.
> > > >
> > > > Is there any possibility to catch the event of new partition
> selection
> > in
> > > > conumserConnector to commit the offsets and data before reconnecting
> to
> > > new
> > > > partitions?
> > > >
> > > > Thanks in advance
> > > > Markus
> > > >
> > > > --
> > > > Markus Roder
> > > > Distelweg 4
> > > > 97318 Kitzingen
> > > > Mail: [EMAIL PROTECTED]
> > > > Profil: http://gplus.to/markusroder
> > > >
> > >
> >
> >
> >
> > --
> > Markus Roder
> > Distelweg 4
> > 97318 Kitzingen
> > Mail: [EMAIL PROTECTED]
> > Profil: http://gplus.to/markusroder
> >
>

--
Markus Roder
Distelweg 4
97318 Kitzingen
Mail: [EMAIL PROTECTED]
Profil: http://gplus.to/markusroder