Hi,
We know in Kafka 0.7, we can specify zk.connect. And with zookeeper, the producer can dynamically detect broker. But in Kafka 0.8, we can't specify zk.connect for producer. How does the producer in Kafka 0.8 auto detect broker?
I have done two experiments.
In first one, I configure the broker.list as broker1 and broker2, and set num.partitions=4 for all brokers. Then, I start broker 1 and broker 3. Without create topic1, I produce message directly, there are 4 partitions generated. Two in broker 1 and two in broker 3. How can the producer detect broker 3?
In second one, I configure the broker.list as broker 1 and broker2, and set num.partitions=4 for all brokers. Then I start broker 3 only. Without create topic2, I produce message directly. As a result ,there is no partition created for topic 2, and the producer report ConnectException, it says fetching topic metadata for topics [Set(topic 2)] from broker 3 failed (kafka.client.ClientUtils$)
Can I guess that the auto detecting ability for producer is weakened in Kafka 0.8? Why you get rid of zookeeper from producer in Kafka 0.8?
Many thanks and best regards!
 

> Can I guess that the auto detecting ability for producer is weakened in
> Kafka 0.8? Why you get rid of zookeeper from producer in Kafka 0.8?
In 0.8, we got rid of zookeeper from the producer and replaced its
functionality with a metadata API. On startup and whenever a request fails,
the producer refreshes its view of the cluster by sending a metadata
request to any of the brokers. So if a broker goes down or a new broker
comes up, the leader for some partitions might change and the producer will
know since its requests to the older leaders will fail.

By removing the dependency of zookeeper from the producer, we have in fact
helped scale the number of producers you can have in a cluster since now
each of them don't need to have a connection to zookeeper and deal with
session expirations when there are minor glitches on the zookeeper cluster.
Another goal of this change was also to enable writing non-java clients
easier.

Thanks,
Neha

 

Hi,
How is the consumer rebalancing works?
In the design part of the official document, it says each consumer decides which partitions it should own. But in below document
https://cwiki.apache.org/confluence/display/KAFKA/Central+Consumer+Coordination
It says there will be a centralized consumer coordinator.
I want to ask about, whether the centralized consumer coordinator is in the Kafka 0.8 code or not?
Many thanks and best regards!
 

That's an experiment for a future release. The consumer logic in 0.8 is
more or less the same as 0.7, ie, running in a decentralized way.

Thanks,

Jun

On Tue, Jan 15, 2013 at 12:10 AM, gj1989lh <[EMAIL PROTECTED]> wrote:

> Hi,
> How is the consumer rebalancing works?
> In the design part of the official document, it says each consumer decides
> which partitions it should own. But in below document
>
> https://cwiki.apache.org/confluence/display/KAFKA/Central+Consumer+Coordination
> It says there will be a centralized consumer coordinator.
> I want to ask about, whether the centralized consumer coordinator is in
> the Kafka 0.8 code or not?
>
>
> Many thanks and best regards!
>