Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - How to use gevent with python


Copy link to this message
-
Re: How to use gevent with python
David Montgomery 2013-11-10, 20:51
I am ready to use low level..........

I do have a gevent worker server that can export that data I need to send
as json.  What about command line?  I do over 10K qps can will be at 25K
soon.

Thanks
On Mon, Nov 11, 2013 at 4:25 AM, Marc Labbe <[EMAIL PROTECTED]> wrote:

> @Kane, Let's be clear, you could send multiple consumer requests in
> parallel, but in terms of handling events, it would make things a little
> harder to manage. E.g. the importance of reading messages in order, are you
> ready to handle duplicates, ... For consumers, gevent may be useful when
> handling events (store in db, forward to other processes, ...) but for
> consumers purely reading from kafka, the model is pretty much to be blocked
> until data becomes available, handle data, read the next batch, and so on.
>
> @David I think you are out of luck for out of the box options for gevent.
> You may be able to do something if you are ready to use low level options.
>
>
> On Sun, Nov 10, 2013 at 2:53 PM, David Montgomery <
> [EMAIL PROTECTED]
> > wrote:
>
> > I am using kafka0. 7.x I should mention and is nit by choice because
> druid,
> > the app I am using only supports 0.7.x. Brod is failing with gevent with
> > 0.7.2.  Brod totally does not work with gevent.
> >
> > Thanks
> >
> >
> >
> >
> >
> > On Mon, Nov 11, 2013 at 3:06 AM, Marc Labbe <[EMAIL PROTECTED]> wrote:
> >
> > > Hi David,
> > >
> > > check for mahendra's fork of kafka-python, he has implemented gevent
> > > support in a branch (
> > https://github.com/mahendra/kafka-python/tree/gevent)
> > > but
> > > it hasn't made it to the main repo (yet). I am not sure why and I
> haven't
> > > tested it myself.
> > >
> > > For brod, AFAIK, it doesn't support 0.8 so you have to use it with
> 0.7.x.
> > > On the other hand, kafka-python is the only python library I know of
> that
> > > supports 0.8.
> > >
> > > cheers,
> > > marc
> > >
> > >
> > > On Sat, Nov 9, 2013 at 6:34 PM, David Montgomery
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > Hi,
> > > >
> > > > Even even further...using/kafka-python (
> > > > https://github.com/mumrah/kafka-python) I get the below error.
>  There
> > is
> > > > nothing wrong with my kafka server.  I can telnet to 9092 just fine.
> > > >
> > > > Again..hoping for a python client that works with gevent.
>  kafka-python
> > > did
> > > > nok event make it past the connection.
> > > >
> > > > kafka = KafkaClient(kafka_domain, 9092)
> > > > No handlers could be found for logger "kafka"
> > > > Traceback (most recent call last):
> > > >   File
> "/home/ubuntu/workspace/rtbhui-devops/servers/worker_server.py",
> > > > line 133, in <module>
> > > >     kafka = KafkaClient(kafka_domain, 9092)
> > > >   File
> > > >
> > > >
> > >
> >
> "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py",
> > > > line 32, in __init__
> > > >     self._load_metadata_for_topics()
> > > >   File
> > > >
> > > >
> > >
> >
> "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py",
> > > > line 69, in _load_metadata_for_topics
> > > >     raise Exception("All servers failed to process request")
> > > > Exception: All servers failed to process request
> > > >
> > > >
> > > >
> > > > On Sun, Nov 10, 2013 at 5:47 AM, David Montgomery <
> > > > [EMAIL PROTECTED]
> > > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I asked this question on StackOverflow.  No response.  No solution
> > > seems
> > > > > tp work...even for a python lib that was designed for gevent and
> > kafka,
> > > >  Is
> > > > > there any otehr libary that works with gevent wit python that
> works?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > am trying to use gevent to write to kafka using brod on python 2.7.
> > > > >
> > > > > Here is the error message I get. Guess its to do blocking. brod
> > > supports
> > > > > tornado but I use gevent.
> > > > >
> > > > > No handlers could be found for logger "brod.socket"Traceback (most
> > >