Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> How to use gevent with python


Copy link to this message
-
Re: How to use gevent with python
New client rewrite proposal includes async Producer, but not async
Consumer, i think there is a reason. You can not send new consume
request before previous one is finished?

On Sun, Nov 10, 2013 at 11:42 AM, Marc Labbe <[EMAIL PROTECTED]> wrote:
> Kane, you can probably achieve async consumer using the client directly
> (kafka.client.send_fetch_request) which a parameter to pass in a callback.
> I haven't used it myself.
>
>
> On Sun, Nov 10, 2013 at 2:33 PM, Marc Labbe <[EMAIL PROTECTED]> wrote:
>
>> I was just browsing in the issues list and found the original issue from
>> mahendra for gevent support,
>> https://github.com/mumrah/kafka-python/issues/37. I suggest moving the
>> discussion there.
>>
>>
>> On Sun, Nov 10, 2013 at 2:06 PM, Marc Labbe <[EMAIL PROTECTED]> wrote:
>>
>>> Hi David,
>>>
>>> check for mahendra's fork of kafka-python, he has implemented gevent
>>> support in a branch (https://github.com/mahendra/kafka-python/tree/gevent) but
>>> it hasn't made it to the main repo (yet). I am not sure why and I haven't
>>> tested it myself.
>>>
>>> For brod, AFAIK, it doesn't support 0.8 so you have to use it with 0.7.x.
>>> On the other hand, kafka-python is the only python library I know of that
>>> supports 0.8.
>>>
>>> cheers,
>>> marc
>>>
>>>
>>> On Sat, Nov 9, 2013 at 6:34 PM, David Montgomery <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> Even even further...using/kafka-python (
>>>> https://github.com/mumrah/kafka-python) I get the below error.  There is
>>>> nothing wrong with my kafka server.  I can telnet to 9092 just fine.
>>>>
>>>> Again..hoping for a python client that works with gevent.  kafka-python
>>>> did
>>>> nok event make it past the connection.
>>>>
>>>> kafka = KafkaClient(kafka_domain, 9092)
>>>> No handlers could be found for logger "kafka"
>>>> Traceback (most recent call last):
>>>>   File "/home/ubuntu/workspace/rtbhui-devops/servers/worker_server.py",
>>>> line 133, in <module>
>>>>     kafka = KafkaClient(kafka_domain, 9092)
>>>>   File
>>>>
>>>> "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py",
>>>> line 32, in __init__
>>>>     self._load_metadata_for_topics()
>>>>   File
>>>>
>>>> "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py",
>>>> line 69, in _load_metadata_for_topics
>>>>     raise Exception("All servers failed to process request")
>>>> Exception: All servers failed to process request
>>>>
>>>>
>>>>
>>>> On Sun, Nov 10, 2013 at 5:47 AM, David Montgomery <
>>>> [EMAIL PROTECTED]
>>>> > wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > I asked this question on StackOverflow.  No response.  No solution
>>>> seems
>>>> > tp work...even for a python lib that was designed for gevent and
>>>> kafka,  Is
>>>> > there any otehr libary that works with gevent wit python that works?
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > am trying to use gevent to write to kafka using brod on python 2.7.
>>>> >
>>>> > Here is the error message I get. Guess its to do blocking. brod
>>>> supports
>>>> > tornado but I use gevent.
>>>> >
>>>> > No handlers could be found for logger "brod.socket"Traceback (most
>>>> recent call last):
>>>> >   File "/var/chef/cache/src/gevent/gevent/greenlet.py", line 328, in
>>>> run
>>>> >     result = self._run(*self.args, **self.kwargs)
>>>> >   File "worker_server.py", line 204, in execute_kafka_pipe
>>>> >     kafka.produce(topic,payload)
>>>> >   File
>>>> "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/base.py",
>>>> line 287, in produce
>>>> >     return self._write(request, callback)
>>>> >   File
>>>> "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/blocking.py",
>>>> line 98, in _write
>>>> >     return self._write(data, callback, retries)
>>>> >   File
>>>> "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/blocking.py",
>>>> line 89, in _write
>>>> >     wrote_length += self._socket.send(data)
>>>> >   File "/var/chef/cache/src/gevent/gevent/socket.py", line 441, in send

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB