Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis


+
Jun Rao 2013-01-15, 05:53
+
Neha Narkhede 2013-01-15, 05:59
+
Jun Rao 2013-01-15, 15:07
Copy link to this message
-
Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis
Hey Guys,

Correct me if I'm wrong, but I believe in 0.8, the producer uses a
metadata request to get topic/partition mappings from the broker. The
broker then interacts with ZK (rather than having the producer do it using
zk.connect).

In the event that the master for a topic/partition fails, a new master
broker will be elected for a given topic/partition pair. When the producer
tries to send to the old broker (which is either dead, or a slave now),
the broker will either not respond, or the response will contain an error
code. In either case, I think the producer will do a new metadata request
(to a broker) to get the latest topic/partition to broker mappings. In
this way, it avoids having to use ZooKeeper: it offloads all ZK work to
the broker.

Cheers,
Chris

On 1/14/13 9:35 PM, "Jun Guo -X (jungu - CIIC at Cisco)" <[EMAIL PROTECTED]>
wrote:

>Hi,
>We know, in kafka 0.8, producer connect to broker directly, it without
>connecting to zookeeper. Than how it achieve zookeeper-based load balance
>on per-request basis?
>Actually, when a topic be created, its partition will distributed in one
>or more brokers. When a message be sent, it will be delivered to a
>certain partition according to its key word. That is to say ,a certain
>must be sent to a fixed partition on a fixed broker. How the so called
>load balancing works?
>
>Best Regards
 
+
Neha Narkhede 2013-01-15, 19:30
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB