Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Leader doesn't get assigned for new topics


+
Rajasekar Elango 2013-09-13, 05:47
+
Neha Narkhede 2013-09-13, 16:07
+
Jun Rao 2013-09-13, 16:09
+
Rajasekar Elango 2013-09-13, 17:02
+
Neha Narkhede 2013-09-13, 17:04
+
Rajasekar Elango 2013-09-13, 21:50
+
Neha Narkhede 2013-09-13, 22:27
+
Rajasekar Elango 2013-09-17, 19:27
+
Neha Narkhede 2013-09-18, 00:06
+
Rajasekar Elango 2013-09-18, 17:31
Copy link to this message
-
Re: Leader doesn't get assigned for new topics
Neha Narkhede 2013-09-18, 21:21
When new topics are created, it takes some time for the controller to
communicate the partition assignment for the new topic to all brokers.
Until that happens, attempts to send/receive data to the brokers fails with
the following kind of error -

[2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
correlation id 622369865 from client  on partition [jmx,3] failed due to
Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)

But eventually the broker receives the updated metadata for the new topic
from the controller. When that happens, these errors go away.
This can also happen on a newly restarted broker if it was not shutdown
using controlled shutdown.

Thanks,
Neha

On Wed, Sep 18, 2013 at 10:30 AM, Rajasekar Elango
<[EMAIL PROTECTED]>wrote:

> From the output of StateChangeLogMerger tool, I see only this error
> repeated;
>
> [2013-09-18 14:16:48,358] ERROR [KafkaApi-1] Error while fetching metadata
> for partition [FunnelProto,0] (kafka.server.KafkaApis)
>
> On the state-change.log itself, I see this error:
>
> [2013-09-18 14:22:48,954] ERROR Conditional update of path
> /brokers/topics/test-1379439240191/partitions/2/state with data {
> "controller_epoch":10, "isr":[ 1, 5, 4 ], "leader":1, "leader_epoch":4,
> "version":1 } and expected version 8 fai
> led due to org.apache.zookeeper.KeeperException$BadVersionException:
> KeeperErrorCode = BadVersion for
> /brokers/topics/test-1379439240191/partitions/2/state
> (kafka.utils.ZkUtils$)
>
> Do you know reason for above error..? Also this problem seem to be
> intermittent, it started working now without any changes. I will continue
> to monitor.
>
> Thanks,
> Raja.
>
>
> On Tue, Sep 17, 2013 at 7:59 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Raja,
> >
> > Could you run the StateChangeLogMerger tool and give it one
> topic-partition
> > that has the above mentioned problem. This tool is documented here -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-7.StateChangeLogMergerTool
> > .
> >
> > Let me know if you run into any issues while using it.
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Sep 17, 2013 at 12:27 PM, Rajasekar Elango
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Neha/Jun,
> > >
> > > The same problem started happening again although now our zookeeper
> > cluster
> > > is configured correctly. The produce always failed with
> > > LeaderNotAvailableException and list topics shows topic is created with
> > > leader "none". In the controller and stage-change log, I am seeing lot
> of
> > > these failures..
> > >
> > >
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [FunnelProto,6]
> failed
> > > due to Partition [FunnelProto,6] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [internal_metrics,3]
> > > failed due to Partition [internal_metrics,3] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [FunnelProto,0]
> failed
> > > due to Partition [FunnelProto,0] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [jmx,3] failed due
> to
> > > Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [FunnelProto,5]
> failed
> > > due to Partition [FunnelProto,5] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > >
> > >
> > > When I ran listTopics command for one of above topic, all partitions
> are
> > > under replicated (we have replication factor set to 3). Any clues on
> what
> > > could be issue and how can we get it back to working?