Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - How to use gevent with python


Copy link to this message
-
How to use gevent with python
David Montgomery 2013-11-09, 21:47
Hi,

I asked this question on StackOverflow.  No response.  No solution seems tp
work...even for a python lib that was designed for gevent and kafka,  Is
there any otehr libary that works with gevent wit python that works?
am trying to use gevent to write to kafka using brod on python 2.7.

Here is the error message I get. Guess its to do blocking. brod supports
tornado but I use gevent.

No handlers could be found for logger "brod.socket"Traceback (most
recent call last):
  File "/var/chef/cache/src/gevent/gevent/greenlet.py", line 328, in run
    result = self._run(*self.args, **self.kwargs)
  File "worker_server.py", line 204, in execute_kafka_pipe
    kafka.produce(topic,payload)
  File "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/base.py",
line 287, in produce
    return self._write(request, callback)
  File "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/blocking.py",
line 98, in _write
    return self._write(data, callback, retries)
  File "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/blocking.py",
line 89, in _write
    wrote_length += self._socket.send(data)
  File "/var/chef/cache/src/gevent/gevent/socket.py", line 441, in send
    self._wait(self._write_event)
  File "/var/chef/cache/src/gevent/gevent/socket.py", line 292, in _wait
    assert watcher.callback is None, 'This socket is already used by
another greenlet: %r' % (watcher.callback, )AssertionError: This
socket is already used by another greenlet: <bound method
Waiter.switch of <gevent.hub.Waiter object at 0x1dece60>><Greenlet at
0x1d4b9b0: execute_kafka_pipe('topic-spend', '{"enode": 1, "city":
"Cairns", "dl": "en", "wnode)> failed with AssertionError

I tried to use gevent-kakfa but depends on gevent-zookeeper.

When trying to connect to zookeeper i get this message:

Traceback (most recent call last):
  File "/home/ubuntu/workspace/devops/servers/worker_server.py", line
68, in <module>
    framework = gevent_zookeeper.ZookeeperFramework('localhost:2181', 10)
  File "/usr/local/lib/python2.7/dist-packages/gevent_zookeeper/framework.py",
line 241, in __init__
    self.client = ZookeeperClient(hosts, timeout)
  File "/usr/local/lib/python2.7/dist-packages/gevent_zookeeper/client.py",
line 211, in __init__
    self._event = gevent.core.event(AttributeError: 'module' object
has no attribute 'core'Exception AttributeError: "'ZookeeperClient'
object has no attribute '_event'" in <bound method
ZookeeperClient.__del__ of <gevent_zookeeper.client.ZookeeperClient
object at 0x274ded0>> ignored

Is there not a python lib that I can I write messages using gevent that
works?