Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - no partition leader yet for partition 0 (0.8.0)


Copy link to this message
-
no partition leader yet for partition 0 (0.8.0)
Chris Curtin 2012-11-27, 21:11
Hi,

I noticed several errors when writing to a topic with 5 partitions. It
looks like the data was written to all 3 brokers, but I get the following
errors:

9961 [main] DEBUG kafka.producer.BrokerPartitionInfo  - Metadata for topic
partition [test1, 0] is errornous:
[PartitionMetadata(0,None,WrappedArray(),WrappedArray(),5)]
kafka.common.LeaderNotAvailableException
at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
<snip>

9962 [main] DEBUG kafka.producer.async.DefaultEventHandler  - Getting the
number of broker partitions registered for topic: test1
9963 [main] DEBUG kafka.producer.BrokerPartitionInfo  - Getting broker
partition info for topic test1
9963 [main] DEBUG kafka.producer.BrokerPartitionInfo  - Topic test1
partition 0 does not have a leader yet
9963 [main] DEBUG kafka.producer.BrokerPartitionInfo  - Topic test1
partition 1 has leader 7
9963 [main] DEBUG kafka.producer.BrokerPartitionInfo  - Topic test1
partition 2 has leader 8
9963 [main] DEBUG kafka.producer.BrokerPartitionInfo  - Topic test1
partition 3 has leader 9
9963 [main] DEBUG kafka.producer.BrokerPartitionInfo  - Topic test1
partition 4 has leader 1
9963 [main] DEBUG kafka.producer.async.DefaultEventHandler  - Broker
partitions registered for topic: test1 are 0,1,2,3,4

This happens a lot as I write data to the Broker.

Topic was created with:

-bash-3.2$ ./kafka-create-topic.sh --topic test1 --partition 5 --replica 3
--zookeeper localhost:2181

Doing a list of topics shows an empty list for that partition:

[2012-11-27 16:03:35,604] INFO Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x23b4218eccd000b, negotiated timeout
= 30000 (org.apache.zookeeper.ClientCnxn)
[2012-11-27 16:03:35,607] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
topic: test1
PartitionMetadata(0,None,List(),List(),5)
PartitionMetadata(1,Some(id:7,creatorId:10.121.31.57-1354023708335,host:10.121.31.57,port:9092),List(id:7,creatorId:10.121.31.57-1354023708335,host:10.121.31.57,port:9092,
id:8,creatorId:10.121.31.57-1354023708340,host:10.121.31.57,port:9093,
id:9,creatorId:10.121.31.57-1354023944130,host:10.121.31.57,port:9094),ArrayBuffer(id:7,creatorId:10.121.31.57-1354023708335,host:10.121.31.57,port:9092,
id:8,creatorId:10.121.31.57-1354023708340,host:10.121.31.57,port:9093,
id:9,creatorId:10.121.31.57-1354023944130,host:10.121.31.57,port:9094),0)
PartitionMetadata(2,Some(id:8,creatorId:10.121.31.57-1354023708340,host:10.121.31.57,port:9093),List(id:8,creatorId:10.121.31.57-1354023708340,host:10.121.31.57,port:9093,
id:9,creatorId:10.121.31.57-1354023944130,host:10.121.31.57,port:9094,
id:1,creatorId:10.121.31.55-1354023701351,host:10.121.31.55,port:9092),ArrayBuffer(id:8,creatorId:10.121.31.57-1354023708340,host:10.121.31.57,port:9093,
id:9,creatorId:10.121.31.57-1354023944130,host:10.121.31.57,port:9094,
id:1,creatorId:10.121.31.55-1354023701351,host:10.121.31.55,port:9092),0)
PartitionMetadata(3,Some(id:9,creatorId:10.121.31.57-1354023944130,host:10.121.31.57,port:9094),List(id:9,creatorId:10.121.31.57-1354023944130,host:10.121.31.57,port:9094,
id:1,creatorId:10.121.31.55-1354023701351,host:10.121.31.55,port:9092,
id:2,creatorId:10.121.31.55-1354023701344,host:10.121.31.55,port:9093),ArrayBuffer(id:9,creatorId:10.121.31.57-1354023944130,host:10.121.31.57,port:9094,
id:1,creatorId:10.121.31.55-1354023701351,host:10.121.31.55,port:9092,
id:2,creatorId:10.121.31.55-1354023701344,host:10.121.31.55,port:9093),0)
PartitionMetadata(4,Some(id:1,creatorId:10.121.31.55-1354023701351,host:10.121.31.55,port:9092),List(id:1,creatorId:10.121.31.55-1354023701351,host:10.121.31.55,port:9092,
id:2,creatorId:10.121.31.55-1354023701344,host:10.121.31.55,port:9093,
id:3,creatorId:10.121.31.55-1354023701345,host:10.121.31.55,port:9094),ArrayBuffer(id:1,creatorId:10.121.31.55-1354023701351,host:10.121.31.55,port:9092,
id:2,creatorId:10.121.31.55-1354023701344,host:10.121.31.55,port:9093,
id:3,creatorId:10.121.31.55-1354023701345,host:10.121.31.55,port:9094),0)
[2012-11-27 16:03:36,005] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)

My partitioner logic is doing a simple modulo on the # of partitions passed:

    return (int) (organizationId % a_numPartitions);

Did I miss a step setting up the topics?

Thanks,

Chris
+
Jun Rao 2012-11-28, 05:13
+
Chris Curtin 2013-03-04, 19:21