Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Leader doesn't get assigned for new topics


+
Rajasekar Elango 2013-09-13, 05:47
+
Neha Narkhede 2013-09-13, 16:07
+
Jun Rao 2013-09-13, 16:09
+
Rajasekar Elango 2013-09-13, 17:02
+
Neha Narkhede 2013-09-13, 17:04
+
Rajasekar Elango 2013-09-13, 21:50
+
Neha Narkhede 2013-09-13, 22:27
+
Rajasekar Elango 2013-09-17, 19:27
+
Neha Narkhede 2013-09-18, 00:06
+
Rajasekar Elango 2013-09-18, 17:31
Copy link to this message
-
Re: Leader doesn't get assigned for new topics
When new topics are created, it takes some time for the controller to
communicate the partition assignment for the new topic to all brokers.
Until that happens, attempts to send/receive data to the brokers fails with
the following kind of error -

[2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
correlation id 622369865 from client  on partition [jmx,3] failed due to
Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)

But eventually the broker receives the updated metadata for the new topic
from the controller. When that happens, these errors go away.
This can also happen on a newly restarted broker if it was not shutdown
using controlled shutdown.

Thanks,
Neha

On Wed, Sep 18, 2013 at 10:30 AM, Rajasekar Elango
<[EMAIL PROTECTED]>wrote:

> From the output of StateChangeLogMerger tool, I see only this error
> repeated;
>
> [2013-09-18 14:16:48,358] ERROR [KafkaApi-1] Error while fetching metadata
> for partition [FunnelProto,0] (kafka.server.KafkaApis)
>
> On the state-change.log itself, I see this error:
>
> [2013-09-18 14:22:48,954] ERROR Conditional update of path
> /brokers/topics/test-1379439240191/partitions/2/state with data {
> "controller_epoch":10, "isr":[ 1, 5, 4 ], "leader":1, "leader_epoch":4,
> "version":1 } and expected version 8 fai
> led due to org.apache.zookeeper.KeeperException$BadVersionException:
> KeeperErrorCode = BadVersion for
> /brokers/topics/test-1379439240191/partitions/2/state
> (kafka.utils.ZkUtils$)
>
> Do you know reason for above error..? Also this problem seem to be
> intermittent, it started working now without any changes. I will continue
> to monitor.
>
> Thanks,
> Raja.
>
>
> On Tue, Sep 17, 2013 at 7:59 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Raja,
> >
> > Could you run the StateChangeLogMerger tool and give it one
> topic-partition
> > that has the above mentioned problem. This tool is documented here -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-7.StateChangeLogMergerTool
> > .
> >
> > Let me know if you run into any issues while using it.
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Sep 17, 2013 at 12:27 PM, Rajasekar Elango
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Neha/Jun,
> > >
> > > The same problem started happening again although now our zookeeper
> > cluster
> > > is configured correctly. The produce always failed with
> > > LeaderNotAvailableException and list topics shows topic is created with
> > > leader "none". In the controller and stage-change log, I am seeing lot
> of
> > > these failures..
> > >
> > >
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [FunnelProto,6]
> failed
> > > due to Partition [FunnelProto,6] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [internal_metrics,3]
> > > failed due to Partition [internal_metrics,3] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [FunnelProto,0]
> failed
> > > due to Partition [FunnelProto,0] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [jmx,3] failed due
> to
> > > Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
> > > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > > correlation id 622369865 from client  on partition [FunnelProto,5]
> failed
> > > due to Partition [FunnelProto,5] doesn't exist on 2
> > > (kafka.server.KafkaApis)
> > >
> > >
> > > When I ran listTopics command for one of above topic, all partitions
> are
> > > under replicated (we have replication factor set to 3). Any clues on
> what
> > > could be issue and how can we get it back to working?