Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Producer will pick one of the two brokers, but never the two at same time [0.8]


+
Alexandre Rodrigues 2013-06-12, 15:02
Copy link to this message
-
Re: Producer will pick one of the two brokers, but never the two at same time [0.8]
Jun Rao 2013-06-12, 15:27
If the leaders exist in both brokers, the producer should be able to
connect to both of them, assuming you don't provide any key when sending
the data. Could you try restarting the producer? If there has been broker
failures, it may take topic.metadata.refresh.interval.ms for the producer
to pick up the newly available partitions (see
http://kafka.apache.org/08/configuration.html for details).

Thanks,

Jun
On Wed, Jun 12, 2013 at 8:01 AM, Alexandre Rodrigues <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have a Kafka 0.8 cluster with two nodes connected to three ZKs, with the
> same configuration but the brokerId (one is 0 and the other 1). I created
> three topics A, B and C with 4 partitions and a replication factor of 1. My
> idea was to have 2 partitions per topic in each broker. However, when I
> connect a producer, I can't have both brokers to write at the same time and
> I don't know what's going on.
>
> My server.config has the following entries:
>
> auto.create.topics.enable=true
> num.partitions=2
>
>
> When I run bin/kafka-list-topic.sh --zookeeper localhost:2181   I get the
> following partition leader assignments:
>
> topic: A  partition: 0    leader: 1       replicas: 1     isr: 1
> topic: A  partition: 1    leader: 0       replicas: 0     isr: 0
> topic: A  partition: 2    leader: 1       replicas: 1     isr: 1
> topic: A  partition: 3    leader: 0       replicas: 0     isr: 0
> topic: B partition: 0    leader: 0       replicas: 0     isr: 0
> topic: B partition: 1    leader: 1       replicas: 1     isr: 1
> topic: B partition: 2    leader: 0       replicas: 0     isr: 0
> topic: B partition: 3    leader: 1       replicas: 1     isr: 1
> topic: C      partition: 0    leader: 0       replicas: 0     isr: 0
> topic: C      partition: 1    leader: 1       replicas: 1     isr: 1
> topic: C      partition: 2    leader: 0       replicas: 0     isr: 0
> topic: C      partition: 3    leader: 1       replicas: 1     isr: 1
>
>
> I've forced reassignment using the kafka-reassign-partitions tool with the
> following JSON:
>
> {"partitions":  [
>    {"topic": "A", "partition": 1, "replicas": [0] },
>    {"topic": "A", "partition": 3, "replicas": [0] },
>    {"topic": "A", "partition": 0, "replicas": [1] },
>    {"topic": "A", "partition": 2, "replicas": [1] },
>    {"topic": "B", "partition": 1, "replicas": [0] },
>    {"topic": "B", "partition": 3, "replicas": [0] },
>    {"topic": "B", "partition": 0, "replicas": [1] },
>    {"topic": "B", "partition": 2, "replicas": [1] },
>    {"topic": "C", "partition": 0, "replicas": [0] },
>    {"topic": "C", "partition": 1, "replicas": [1] },
>    {"topic": "C", "partition": 2, "replicas": [0] },
>    {"topic": "C", "partition": 3, "replicas": [1] }
> ]}
>
> After reassignment, I've restarted producer and nothing worked. I've tried
> also to restart both brokers and producer and nothing.
>
> The producer contains this logs:
>
> 2013-06-12 14:48:46,467] WARN Error while fetching metadata    partition 0
>     leader: none    replicas:       isr:    isUnderReplicated: false for
> topic partition [C,0]: [class kafka.common.LeaderNotAvailableException]
> (kafka.producer.BrokerPartitionInfo)
> [2013-06-12 14:48:46,467] WARN Error while fetching metadata    partition 0
>     leader: none    replicas:       isr:    isUnderReplicated: false for
> topic partition [C,0]: [class kafka.common.LeaderNotAvailableException]
> (kafka.producer.BrokerPartitionInfo)
> [2013-06-12 14:48:46,468] WARN Error while fetching metadata    partition 2
>     leader: none    replicas:       isr:    isUnderReplicated: false for
> topic partition [C,2]: [class kafka.common.LeaderNotAvailableException]
> (kafka.producer.BrokerPartitionInfo)
> [2013-06-12 14:48:46,468] WARN Error while fetching metadata    partition 2
>     leader: none    replicas:       isr:    isUnderReplicated: false for
> topic partition [C,2]: [class kafka.common.LeaderNotAvailableException]
> (kafka.producer.BrokerPartitionInfo)
>
>
> And sometimes lines like this:

 
+
Alexandre Rodrigues 2013-06-12, 16:29
+
Jun Rao 2013-06-13, 03:58
+
Alexandre Rodrigues 2013-06-13, 09:16
+
Alexandre Rodrigues 2013-06-13, 11:14
+
Alexandre Rodrigues 2013-06-13, 13:43
+
Jun Rao 2013-06-13, 14:42
+
Alexandre Rodrigues 2013-06-13, 14:51