Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Leader doesn't get assigned for new topics


Copy link to this message
-
Re: Leader doesn't get assigned for new topics
From the output of StateChangeLogMerger tool, I see only this error
repeated;

[2013-09-18 14:16:48,358] ERROR [KafkaApi-1] Error while fetching metadata
for partition [FunnelProto,0] (kafka.server.KafkaApis)

On the state-change.log itself, I see this error:

[2013-09-18 14:22:48,954] ERROR Conditional update of path
/brokers/topics/test-1379439240191/partitions/2/state with data {
"controller_epoch":10, "isr":[ 1, 5, 4 ], "leader":1, "leader_epoch":4,
"version":1 } and expected version 8 fai
led due to org.apache.zookeeper.KeeperException$BadVersionException:
KeeperErrorCode = BadVersion for
/brokers/topics/test-1379439240191/partitions/2/state (kafka.utils.ZkUtils$)

Do you know reason for above error..? Also this problem seem to be
intermittent, it started working now without any changes. I will continue
to monitor.

Thanks,
Raja.
On Tue, Sep 17, 2013 at 7:59 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Raja,
>
> Could you run the StateChangeLogMerger tool and give it one topic-partition
> that has the above mentioned problem. This tool is documented here -
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-7.StateChangeLogMergerTool
> .
>
> Let me know if you run into any issues while using it.
>
> Thanks,
> Neha
>
>
> On Tue, Sep 17, 2013 at 12:27 PM, Rajasekar Elango
> <[EMAIL PROTECTED]>wrote:
>
> > Neha/Jun,
> >
> > The same problem started happening again although now our zookeeper
> cluster
> > is configured correctly. The produce always failed with
> > LeaderNotAvailableException and list topics shows topic is created with
> > leader "none". In the controller and stage-change log, I am seeing lot of
> > these failures..
> >
> >
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [FunnelProto,6] failed
> > due to Partition [FunnelProto,6] doesn't exist on 2
> > (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [internal_metrics,3]
> > failed due to Partition [internal_metrics,3] doesn't exist on 2
> > (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [FunnelProto,0] failed
> > due to Partition [FunnelProto,0] doesn't exist on 2
> > (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [jmx,3] failed due to
> > Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [FunnelProto,5] failed
> > due to Partition [FunnelProto,5] doesn't exist on 2
> > (kafka.server.KafkaApis)
> >
> >
> > When I ran listTopics command for one of above topic, all partitions are
> > under replicated (we have replication factor set to 3). Any clues on what
> > could be issue and how can we get it back to working?
> >
> > Thanks,
> > Raja.
> >
> >
> >
> > On Fri, Sep 13, 2013 at 6:26 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Ah ok. Thanks for sharing that.
> > >
> > >
> > >
> > > On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > We have 3 zookeeper node in the cluster with a hardware load
> balancer .
> > >  In
> > > > one of the zookeeper, we did not configure ensemble correctly
> (server.n
> > > > property in zoo.cfg) . So it ended up as like 2 nodes in one cluster,
> > one
> > > > node in other cluster. The load balancer is randomly hitting one of 2
> > > > zookeepers in two different cluster.
> > > >
> > > > Thanks,
> > > > Raja.
> > > >
> > > >
> > > > On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Just curious to know, what was the misconfiguration?
> > > >
Thanks,
Raja.