Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - as i understand rebalance happens on client side


Copy link to this message
-
Re: as i understand rebalance happens on client side
Neha Narkhede 2013-10-01, 16:40
Currently we are preparing for the 0.8-final release. But 0.9 is not slated
to release until early next year.

Thanks,
Neha
On Tue, Oct 1, 2013 at 9:26 AM, Kane Kane <[EMAIL PROTECTED]> wrote:

> Btw, is it expected to be released on Oct 31?
>
> Thanks!
>
>
> On Tue, Oct 1, 2013 at 9:01 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > We do plan to move the group membership over to the server side and have
> a
> > very thin consumer client. The proposal is here -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPIand
> > this is being planned for the 0.9 release. Once this is complete, the
> > non-java clients can easily leverage the group management feature.
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Oct 1, 2013 at 8:56 AM, Kane Kane <[EMAIL PROTECTED]> wrote:
> >
> > > The reason i was asking is that this library seems to have support only
> > for
> > > SimpleConsumer https://github.com/mumrah/kafka-python/, i was curious
> if
> > > all should be implemented on client or kafka has some rebalancing logic
> > and
> > > prevent consuming from the same queue on server side in case of
> > > SimpleConsumer api, but I see now that everything should implemented on
> > the
> > > client side.
> > >
> > > Thanks.
> > >
> > >
> > > On Tue, Oct 1, 2013 at 8:52 AM, Guozhang Wang <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > I do not understand your question, what are you trying to implement?
> > > >
> > > >
> > > > On Tue, Oct 1, 2013 at 8:42 AM, Kane Kane <[EMAIL PROTECTED]>
> > wrote:
> > > >
> > > > > So essentially you can't do "queue" pattern, unless you somehow
> > > implement
> > > > > locking on the client?
> > > > >
> > > > >
> > > > > On Tue, Oct 1, 2013 at 8:35 AM, Guozhang Wang <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > SimpleConsumer do not have any concept of group management, only
> > the
> > > > > > high-level consumers have. So multiple simple consumers can
> > > > independently
> > > > > > consume from the same partition(s).
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 1, 2013 at 8:11 AM, Kane Kane <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > > >
> > > > > > > Yeah, I noticed that, i'm curious how balancing happens if
> > > > > SimpleConsumer
> > > > > > > is used. I.e. i can provide a partition to read from if i use
> > > > > > > SimpleConsumer, but what if someone else already attached to
> that
> > > > > > > partition, what would happen? Also what would happen if one
> > > > > > SimpleConsumer
> > > > > > > attached to all partitions? Noone would be able to join?
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Oct 1, 2013 at 6:33 AM, Neha Narkhede <
> > > > [EMAIL PROTECTED]
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > There are 2 types of consumer clients in Kafka -
> > > > > > > ZookeeperConsumerConnector
> > > > > > > > and SimpleConsumer. Only the former has the re balancing
> logic.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Neha
> > > > > > > > On Oct 1, 2013 6:30 AM, "Kane Kane" <[EMAIL PROTECTED]>
> > > wrote:
> > > > > > > >
> > > > > > > > > But it looks like some clients don't implement it?
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
>