Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Leader doesn't get assigned for new topics


Copy link to this message
-
Re: Leader doesn't get assigned for new topics
From the output of StateChangeLogMerger tool, I see only this error
repeated;

[2013-09-18 14:16:48,358] ERROR [KafkaApi-1] Error while fetching metadata
for partition [FunnelProto,0] (kafka.server.KafkaApis)

On the state-change.log itself, I see this error:

[2013-09-18 14:22:48,954] ERROR Conditional update of path
/brokers/topics/test-1379439240191/partitions/2/state with data {
"controller_epoch":10, "isr":[ 1, 5, 4 ], "leader":1, "leader_epoch":4,
"version":1 } and expected version 8 fai
led due to org.apache.zookeeper.KeeperException$BadVersionException:
KeeperErrorCode = BadVersion for
/brokers/topics/test-1379439240191/partitions/2/state (kafka.utils.ZkUtils$)

Do you know reason for above error..? Also this problem seem to be
intermittent, it started working now without any changes. I will continue
to monitor.

Thanks,
Raja.
On Tue, Sep 17, 2013 at 7:59 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Raja,
>
> Could you run the StateChangeLogMerger tool and give it one topic-partition
> that has the above mentioned problem. This tool is documented here -
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-7.StateChangeLogMergerTool
> .
>
> Let me know if you run into any issues while using it.
>
> Thanks,
> Neha
>
>
> On Tue, Sep 17, 2013 at 12:27 PM, Rajasekar Elango
> <[EMAIL PROTECTED]>wrote:
>
> > Neha/Jun,
> >
> > The same problem started happening again although now our zookeeper
> cluster
> > is configured correctly. The produce always failed with
> > LeaderNotAvailableException and list topics shows topic is created with
> > leader "none". In the controller and stage-change log, I am seeing lot of
> > these failures..
> >
> >
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [FunnelProto,6] failed
> > due to Partition [FunnelProto,6] doesn't exist on 2
> > (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [internal_metrics,3]
> > failed due to Partition [internal_metrics,3] doesn't exist on 2
> > (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [FunnelProto,0] failed
> > due to Partition [FunnelProto,0] doesn't exist on 2
> > (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [jmx,3] failed due to
> > Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
> > [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> > correlation id 622369865 from client  on partition [FunnelProto,5] failed
> > due to Partition [FunnelProto,5] doesn't exist on 2
> > (kafka.server.KafkaApis)
> >
> >
> > When I ran listTopics command for one of above topic, all partitions are
> > under replicated (we have replication factor set to 3). Any clues on what
> > could be issue and how can we get it back to working?
> >
> > Thanks,
> > Raja.
> >
> >
> >
> > On Fri, Sep 13, 2013 at 6:26 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Ah ok. Thanks for sharing that.
> > >
> > >
> > >
> > > On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > We have 3 zookeeper node in the cluster with a hardware load
> balancer .
> > >  In
> > > > one of the zookeeper, we did not configure ensemble correctly
> (server.n
> > > > property in zoo.cfg) . So it ended up as like 2 nodes in one cluster,
> > one
> > > > node in other cluster. The load balancer is randomly hitting one of 2
> > > > zookeepers in two different cluster.
> > > >
> > > > Thanks,
> > > > Raja.
> > > >
> > > >
> > > > On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Just curious to know, what was the misconfiguration?
> > > >
Thanks,
Raja.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB