Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> as i understand rebalance happens on client side


Copy link to this message
-
Re: as i understand rebalance happens on client side
We do plan to move the group membership over to the server side and have a
very thin consumer client. The proposal is here -
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPIand
this is being planned for the 0.9 release. Once this is complete, the
non-java clients can easily leverage the group management feature.

Thanks,
Neha
On Tue, Oct 1, 2013 at 8:56 AM, Kane Kane <[EMAIL PROTECTED]> wrote:

> The reason i was asking is that this library seems to have support only for
> SimpleConsumer https://github.com/mumrah/kafka-python/, i was curious if
> all should be implemented on client or kafka has some rebalancing logic and
> prevent consuming from the same queue on server side in case of
> SimpleConsumer api, but I see now that everything should implemented on the
> client side.
>
> Thanks.
>
>
> On Tue, Oct 1, 2013 at 8:52 AM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
>
> > I do not understand your question, what are you trying to implement?
> >
> >
> > On Tue, Oct 1, 2013 at 8:42 AM, Kane Kane <[EMAIL PROTECTED]> wrote:
> >
> > > So essentially you can't do "queue" pattern, unless you somehow
> implement
> > > locking on the client?
> > >
> > >
> > > On Tue, Oct 1, 2013 at 8:35 AM, Guozhang Wang <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > SimpleConsumer do not have any concept of group management, only the
> > > > high-level consumers have. So multiple simple consumers can
> > independently
> > > > consume from the same partition(s).
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Oct 1, 2013 at 8:11 AM, Kane Kane <[EMAIL PROTECTED]>
> > wrote:
> > > >
> > > > > Yeah, I noticed that, i'm curious how balancing happens if
> > > SimpleConsumer
> > > > > is used. I.e. i can provide a partition to read from if i use
> > > > > SimpleConsumer, but what if someone else already attached to that
> > > > > partition, what would happen? Also what would happen if one
> > > > SimpleConsumer
> > > > > attached to all partitions? Noone would be able to join?
> > > > >
> > > > >
> > > > > On Tue, Oct 1, 2013 at 6:33 AM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > There are 2 types of consumer clients in Kafka -
> > > > > ZookeeperConsumerConnector
> > > > > > and SimpleConsumer. Only the former has the re balancing logic.
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > > On Oct 1, 2013 6:30 AM, "Kane Kane" <[EMAIL PROTECTED]>
> wrote:
> > > > > >
> > > > > > > But it looks like some clients don't implement it?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>