Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis


Copy link to this message
-
Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis
Chris Riccomini 2013-01-15, 19:26
Hey Guys,

Correct me if I'm wrong, but I believe in 0.8, the producer uses a
metadata request to get topic/partition mappings from the broker. The
broker then interacts with ZK (rather than having the producer do it using
zk.connect).

In the event that the master for a topic/partition fails, a new master
broker will be elected for a given topic/partition pair. When the producer
tries to send to the old broker (which is either dead, or a slave now),
the broker will either not respond, or the response will contain an error
code. In either case, I think the producer will do a new metadata request
(to a broker) to get the latest topic/partition to broker mappings. In
this way, it avoids having to use ZooKeeper: it offloads all ZK work to
the broker.

Cheers,
Chris

On 1/14/13 9:35 PM, "Jun Guo -X (jungu - CIIC at Cisco)" <[EMAIL PROTECTED]>
wrote:

>Hi,
>We know, in kafka 0.8, producer connect to broker directly, it without
>connecting to zookeeper. Than how it achieve zookeeper-based load balance
>on per-request basis?
>Actually, when a topic be created, its partition will distributed in one
>or more brokers. When a message be sent, it will be delivered to a
>certain partition according to its key word. That is to say ,a certain
>must be sent to a fixed partition on a fixed broker. How the so called
>load balancing works?
>
>Best Regards