Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # dev - Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis


+
Jun Rao 2013-01-15, 05:53
Copy link to this message
-
Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis
Neha Narkhede 2013-01-15, 05:59
>
>
> Than how it achieve zookeeper-based load balance on per-request basis?
You are asking two related but different questions. One is how does load
balancing work and the other is how does broker discovery works. Jun
explained how load balancing works and how requests are routed to
partitons. In 0.7, there were 2 options for broker discovery - zookeeper
and hardware load balancer (VIP). The zookeeper based producer got notified
by zookeeper whenever a new broker came up or an existing broker went down.
In 0.8, we got rid of zookeeper from the producer and replaced it with a
metadata API. On startup and whenever a request fails, the producer
refreshes its view of the cluster by sending a metadata request to any of
the brokers. So if a broker goes down or a new broker comes up, the leader
for some partitions might change and the producer will know since its
requests to the older leaders will fail.

Hope that helps,
Neha
On Mon, Jan 14, 2013 at 9:52 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Basically, we spread partitions among multiple brokers. If a message is
> sent without a key, the producer picks a random partition to balance the
> load. If a message has a key, the default partitioner hashes the key to one
> of the partitions deterministically. Then, the load may not always be
> balanced.
>
> Thanks,
>
> Jun
>
> On Mon, Jan 14, 2013 at 9:35 PM, Jun Guo -X (jungu - CIIC at Cisco) <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> > We know, in kafka 0.8, producer connect to broker directly, it without
> > connecting to zookeeper. Than how it achieve zookeeper-based load balance
> > on per-request basis?
> > Actually, when a topic be created, its partition will distributed in one
> > or more brokers. When a message be sent, it will be delivered to a
> certain
> > partition according to its key word. That is to say ,a certain must be
> sent
> > to a fixed partition on a fixed broker. How the so called load balancing
> > works?
> >
> > Best Regards
> >
>

 
+
Jun Rao 2013-01-15, 15:07
+
Chris Riccomini 2013-01-15, 19:26
+
Neha Narkhede 2013-01-15, 19:30