Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> How to use gevent with python


Copy link to this message
-
Re: How to use gevent with python
I am ready to use low level..........

I do have a gevent worker server that can export that data I need to send
as json.  What about command line?  I do over 10K qps can will be at 25K
soon.

Thanks
On Mon, Nov 11, 2013 at 4:25 AM, Marc Labbe <[EMAIL PROTECTED]> wrote:

> @Kane, Let's be clear, you could send multiple consumer requests in
> parallel, but in terms of handling events, it would make things a little
> harder to manage. E.g. the importance of reading messages in order, are you
> ready to handle duplicates, ... For consumers, gevent may be useful when
> handling events (store in db, forward to other processes, ...) but for
> consumers purely reading from kafka, the model is pretty much to be blocked
> until data becomes available, handle data, read the next batch, and so on.
>
> @David I think you are out of luck for out of the box options for gevent.
> You may be able to do something if you are ready to use low level options.
>
>
> On Sun, Nov 10, 2013 at 2:53 PM, David Montgomery <
> [EMAIL PROTECTED]
> > wrote:
>
> > I am using kafka0. 7.x I should mention and is nit by choice because
> druid,
> > the app I am using only supports 0.7.x. Brod is failing with gevent with
> > 0.7.2.  Brod totally does not work with gevent.
> >
> > Thanks
> >
> >
> >
> >
> >
> > On Mon, Nov 11, 2013 at 3:06 AM, Marc Labbe <[EMAIL PROTECTED]> wrote:
> >
> > > Hi David,
> > >
> > > check for mahendra's fork of kafka-python, he has implemented gevent
> > > support in a branch (
> > https://github.com/mahendra/kafka-python/tree/gevent)
> > > but
> > > it hasn't made it to the main repo (yet). I am not sure why and I
> haven't
> > > tested it myself.
> > >
> > > For brod, AFAIK, it doesn't support 0.8 so you have to use it with
> 0.7.x.
> > > On the other hand, kafka-python is the only python library I know of
> that
> > > supports 0.8.
> > >
> > > cheers,
> > > marc
> > >
> > >
> > > On Sat, Nov 9, 2013 at 6:34 PM, David Montgomery
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > Hi,
> > > >
> > > > Even even further...using/kafka-python (
> > > > https://github.com/mumrah/kafka-python) I get the below error.
>  There
> > is
> > > > nothing wrong with my kafka server.  I can telnet to 9092 just fine.
> > > >
> > > > Again..hoping for a python client that works with gevent.
>  kafka-python
> > > did
> > > > nok event make it past the connection.
> > > >
> > > > kafka = KafkaClient(kafka_domain, 9092)
> > > > No handlers could be found for logger "kafka"
> > > > Traceback (most recent call last):
> > > >   File
> "/home/ubuntu/workspace/rtbhui-devops/servers/worker_server.py",
> > > > line 133, in <module>
> > > >     kafka = KafkaClient(kafka_domain, 9092)
> > > >   File
> > > >
> > > >
> > >
> >
> "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py",
> > > > line 32, in __init__
> > > >     self._load_metadata_for_topics()
> > > >   File
> > > >
> > > >
> > >
> >
> "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py",
> > > > line 69, in _load_metadata_for_topics
> > > >     raise Exception("All servers failed to process request")
> > > > Exception: All servers failed to process request
> > > >
> > > >
> > > >
> > > > On Sun, Nov 10, 2013 at 5:47 AM, David Montgomery <
> > > > [EMAIL PROTECTED]
> > > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I asked this question on StackOverflow.  No response.  No solution
> > > seems
> > > > > tp work...even for a python lib that was designed for gevent and
> > kafka,
> > > >  Is
> > > > > there any otehr libary that works with gevent wit python that
> works?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > am trying to use gevent to write to kafka using brod on python 2.7.
> > > > >
> > > > > Here is the error message I get. Guess its to do blocking. brod
> > > supports
> > > > > tornado but I use gevent.
> > > > >
> > > > > No handlers could be found for logger "brod.socket"Traceback (most
> > >
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB