Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> as i understand rebalance happens on client side


Copy link to this message
-
Re: as i understand rebalance happens on client side
To be honest, 0.8 it's a pretty big change from how things are currently
managed in samsa. I don't have much of an ETA at the moment, but I'm hoping
to find the time in the next month or two.

Keith.
On Thu, Oct 3, 2013 at 11:59 AM, Kane Kane <[EMAIL PROTECTED]> wrote:

> Hi Keith, thanks for update! Interestingly i've found your library
> yesterday and was going to borrow your zookeeper code for partition
> management. Do you have any ETA for the 0.8 support?
>
> Thanks!
>
>
> On Thu, Oct 3, 2013 at 8:09 AM, Keith Bourgoin <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Kane,
> >
> > I just wanted to chime in as well. I currently maintain samsa (
> > https://github.com/getsamsa/samsa or PyPI), which does implement the
> > rebalancing logic, but currently doesn't support 0.8. If you're on 0.7.x
> > still, it might be helpful. If not, we're working on 0.8 support, but it
> > probably won't be ready for a bit.
> >
> > Cheers,
> >
> > Keith.
> >
> >
> > On Tue, Oct 1, 2013 at 1:39 PM, Kane Kane <[EMAIL PROTECTED]> wrote:
> >
> > > Thanks for reply, David, your library is great and indeed the
> rebalancing
> > > is currently somewhat quirky and complicated. And I guess it doesn't
> make
> > > sense to implement it considering 0.9 is planned relatively soon.
> > >
> > >
> > > On Tue, Oct 1, 2013 at 10:09 AM, David Arthur <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Kane,
> > > >
> > > > I'm the creator of kafka-python, just thought I'd give some insight.
> > > >
> > > > Consumer rebalancing is actually pretty tricky to get right. It
> > requires
> > > > interaction with ZooKeeper which (though possible via kazoo) is
> > something
> > > > I've tried to avoid in kafka-python. It also seems a little strange
> to
> > me
> > > > to mix your consumers between Java/Scala and Python. If you really
> need
> > > > rebalancing between Python consumers, you'd have to implement that on
> > top
> > > > of kafka-python.
> > > >
> > > > Once the coordinator API is finalized for 0.9, I (or someone) will
> work
> > > on
> > > > implementing it in kafka-python
> > > >
> > > > Cheers
> > > > -David
> > > >
> > > >
> > > > On 10/1/13 11:56 AM, Kane Kane wrote:
> > > >
> > > >> The reason i was asking is that this library seems to have support
> > only
> > > >> for
> > > >> SimpleConsumer https://github.com/mumrah/**kafka-python/<
> > > https://github.com/mumrah/kafka-python/>,
> > > >> i was curious if
> > > >> all should be implemented on client or kafka has some rebalancing
> > logic
> > > >> and
> > > >> prevent consuming from the same queue on server side in case of
> > > >> SimpleConsumer api, but I see now that everything should implemented
> > on
> > > >> the
> > > >> client side.
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >> On Tue, Oct 1, 2013 at 8:52 AM, Guozhang Wang <[EMAIL PROTECTED]>
> > > wrote:
> > > >>
> > > >>  I do not understand your question, what are you trying to
> implement?
> > > >>>
> > > >>>
> > > >>> On Tue, Oct 1, 2013 at 8:42 AM, Kane Kane <[EMAIL PROTECTED]>
> > > wrote:
> > > >>>
> > > >>>  So essentially you can't do "queue" pattern, unless you somehow
> > > >>>> implement
> > > >>>> locking on the client?
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Oct 1, 2013 at 8:35 AM, Guozhang Wang <[EMAIL PROTECTED]
> >
> > > >>>>
> > > >>> wrote:
> > > >>>
> > > >>>> SimpleConsumer do not have any concept of group management, only
> the
> > > >>>>> high-level consumers have. So multiple simple consumers can
> > > >>>>>
> > > >>>> independently
> > > >>>
> > > >>>> consume from the same partition(s).
> > > >>>>>
> > > >>>>> Guozhang
> > > >>>>>
> > > >>>>>
> > > >>>>> On Tue, Oct 1, 2013 at 8:11 AM, Kane Kane <[EMAIL PROTECTED]
> >
> > > >>>>>
> > > >>>> wrote:
> > > >>>
> > > >>>> Yeah, I noticed that, i'm curious how balancing happens if
> > > >>>>>>
> > > >>>>> SimpleConsumer
> > > >>>>
> > > >>>>> is used. I.e. i can provide a partition to read from if i use
> > > >>>>>> SimpleConsumer, but what if someone else already attached to