Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> as i understand rebalance happens on client side


Copy link to this message
-
Re: as i understand rebalance happens on client side
To be honest, 0.8 it's a pretty big change from how things are currently
managed in samsa. I don't have much of an ETA at the moment, but I'm hoping
to find the time in the next month or two.

Keith.
On Thu, Oct 3, 2013 at 11:59 AM, Kane Kane <[EMAIL PROTECTED]> wrote:

> Hi Keith, thanks for update! Interestingly i've found your library
> yesterday and was going to borrow your zookeeper code for partition
> management. Do you have any ETA for the 0.8 support?
>
> Thanks!
>
>
> On Thu, Oct 3, 2013 at 8:09 AM, Keith Bourgoin <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Kane,
> >
> > I just wanted to chime in as well. I currently maintain samsa (
> > https://github.com/getsamsa/samsa or PyPI), which does implement the
> > rebalancing logic, but currently doesn't support 0.8. If you're on 0.7.x
> > still, it might be helpful. If not, we're working on 0.8 support, but it
> > probably won't be ready for a bit.
> >
> > Cheers,
> >
> > Keith.
> >
> >
> > On Tue, Oct 1, 2013 at 1:39 PM, Kane Kane <[EMAIL PROTECTED]> wrote:
> >
> > > Thanks for reply, David, your library is great and indeed the
> rebalancing
> > > is currently somewhat quirky and complicated. And I guess it doesn't
> make
> > > sense to implement it considering 0.9 is planned relatively soon.
> > >
> > >
> > > On Tue, Oct 1, 2013 at 10:09 AM, David Arthur <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Kane,
> > > >
> > > > I'm the creator of kafka-python, just thought I'd give some insight.
> > > >
> > > > Consumer rebalancing is actually pretty tricky to get right. It
> > requires
> > > > interaction with ZooKeeper which (though possible via kazoo) is
> > something
> > > > I've tried to avoid in kafka-python. It also seems a little strange
> to
> > me
> > > > to mix your consumers between Java/Scala and Python. If you really
> need
> > > > rebalancing between Python consumers, you'd have to implement that on
> > top
> > > > of kafka-python.
> > > >
> > > > Once the coordinator API is finalized for 0.9, I (or someone) will
> work
> > > on
> > > > implementing it in kafka-python
> > > >
> > > > Cheers
> > > > -David
> > > >
> > > >
> > > > On 10/1/13 11:56 AM, Kane Kane wrote:
> > > >
> > > >> The reason i was asking is that this library seems to have support
> > only
> > > >> for
> > > >> SimpleConsumer https://github.com/mumrah/**kafka-python/<
> > > https://github.com/mumrah/kafka-python/>,
> > > >> i was curious if
> > > >> all should be implemented on client or kafka has some rebalancing
> > logic
> > > >> and
> > > >> prevent consuming from the same queue on server side in case of
> > > >> SimpleConsumer api, but I see now that everything should implemented
> > on
> > > >> the
> > > >> client side.
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >> On Tue, Oct 1, 2013 at 8:52 AM, Guozhang Wang <[EMAIL PROTECTED]>
> > > wrote:
> > > >>
> > > >>  I do not understand your question, what are you trying to
> implement?
> > > >>>
> > > >>>
> > > >>> On Tue, Oct 1, 2013 at 8:42 AM, Kane Kane <[EMAIL PROTECTED]>
> > > wrote:
> > > >>>
> > > >>>  So essentially you can't do "queue" pattern, unless you somehow
> > > >>>> implement
> > > >>>> locking on the client?
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Oct 1, 2013 at 8:35 AM, Guozhang Wang <[EMAIL PROTECTED]
> >
> > > >>>>
> > > >>> wrote:
> > > >>>
> > > >>>> SimpleConsumer do not have any concept of group management, only
> the
> > > >>>>> high-level consumers have. So multiple simple consumers can
> > > >>>>>
> > > >>>> independently
> > > >>>
> > > >>>> consume from the same partition(s).
> > > >>>>>
> > > >>>>> Guozhang
> > > >>>>>
> > > >>>>>
> > > >>>>> On Tue, Oct 1, 2013 at 8:11 AM, Kane Kane <[EMAIL PROTECTED]
> >
> > > >>>>>
> > > >>>> wrote:
> > > >>>
> > > >>>> Yeah, I noticed that, i'm curious how balancing happens if
> > > >>>>>>
> > > >>>>> SimpleConsumer
> > > >>>>
> > > >>>>> is used. I.e. i can provide a partition to read from if i use
> > > >>>>>> SimpleConsumer, but what if someone else already attached to

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB