Kafka, mail # user - Re: Consumer re-design and Python - 2013-02-11, 15:32
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: Consumer re-design and Python
On 1/31/13 3:30 PM, Marc Labbe wrote:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design

and

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design

are the current design docs (as far as I know).
 From my understanding, the client redesign has not been finalized and
it still in-progress/todo.
With 0.8, you will not need to connect to ZooKeeper from the clients.
With KAFKA-657, offsets are centrally managed by the broker. Any broker
can handle these requests.
I am working updating my Python client:
https://github.com/mumrah/kafka-python, still a ways to go yet. The
biggest change (besides centralized offset management) is that each
topic+partition is owned by a specific broker (the leader). When
producing messages, you must send them to the correct leader. This
requires that clients maintain some state of what belongs where which is
a pain, but such is the cost of replication.
-David
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB