Yeah, I noticed that, i'm curious how balancing happens if SimpleConsumer is used. I.e. i can provide a partition to read from if i use SimpleConsumer, but what if someone else already attached to that partition, what would happen? Also what would happen if one SimpleConsumer attached to all partitions? Noone would be able to join? On Tue, Oct 1, 2013 at 6:33 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
I think you want locking to prevent two processes from consuming the same partition. You can just use the high level consumer to get this group management functionality. It ensures that a partition is only consumed by one high level consumer at any given time. Is there any particular reason why you want to use SimpleConsumer instead?
Thanks, Neha On Tue, Oct 1, 2013 at 8:42 AM, Kane Kane <[EMAIL PROTECTED]> wrote:
Yes, and I actually want to use high level api, i just didn't understand how kafka works inside. For some reason i thought in case of SimpleConsumer, kafka would implement locking and rebalancing on server side, letting client know that it can't attach to specific partition, etc. But i see now that all management should happen on the client.
Thanks. On Tue, Oct 1, 2013 at 8:48 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
The reason i was asking is that this library seems to have support only for SimpleConsumer https://github.com/mumrah/kafka-python/, i was curious if all should be implemented on client or kafka has some rebalancing logic and prevent consuming from the same queue on server side in case of SimpleConsumer api, but I see now that everything should implemented on the client side.
Thanks. On Tue, Oct 1, 2013 at 8:52 AM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
Thanks for reply, David, your library is great and indeed the rebalancing is currently somewhat quirky and complicated. And I guess it doesn't make sense to implement it considering 0.9 is planned relatively soon. On Tue, Oct 1, 2013 at 10:09 AM, David Arthur <[EMAIL PROTECTED]> wrote:
I just wanted to chime in as well. I currently maintain samsa ( https://github.com/getsamsa/samsa or PyPI), which does implement the rebalancing logic, but currently doesn't support 0.8. If you're on 0.7.x still, it might be helpful. If not, we're working on 0.8 support, but it probably won't be ready for a bit.
Keith. On Tue, Oct 1, 2013 at 1:39 PM, Kane Kane <[EMAIL PROTECTED]> wrote:
To be honest, 0.8 it's a pretty big change from how things are currently managed in samsa. I don't have much of an ETA at the moment, but I'm hoping to find the time in the next month or two.
Keith. On Thu, Oct 3, 2013 at 11:59 AM, Kane Kane <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext