Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Leader doesn't get assigned for new topics


+
Rajasekar Elango 2013-09-13, 05:47
+
Neha Narkhede 2013-09-13, 16:07
+
Jun Rao 2013-09-13, 16:09
+
Rajasekar Elango 2013-09-13, 17:02
+
Neha Narkhede 2013-09-13, 17:04
+
Rajasekar Elango 2013-09-13, 21:50
+
Neha Narkhede 2013-09-13, 22:27
+
Rajasekar Elango 2013-09-17, 19:27
Copy link to this message
-
Re: Leader doesn't get assigned for new topics
Raja,

Could you run the StateChangeLogMerger tool and give it one topic-partition
that has the above mentioned problem. This tool is documented here -
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-7.StateChangeLogMergerTool
.

Let me know if you run into any issues while using it.

Thanks,
Neha
On Tue, Sep 17, 2013 at 12:27 PM, Rajasekar Elango
<[EMAIL PROTECTED]>wrote:

> Neha/Jun,
>
> The same problem started happening again although now our zookeeper cluster
> is configured correctly. The produce always failed with
> LeaderNotAvailableException and list topics shows topic is created with
> leader "none". In the controller and stage-change log, I am seeing lot of
> these failures..
>
>
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client  on partition [FunnelProto,6] failed
> due to Partition [FunnelProto,6] doesn't exist on 2
> (kafka.server.KafkaApis)
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client  on partition [internal_metrics,3]
> failed due to Partition [internal_metrics,3] doesn't exist on 2
> (kafka.server.KafkaApis)
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client  on partition [FunnelProto,0] failed
> due to Partition [FunnelProto,0] doesn't exist on 2
> (kafka.server.KafkaApis)
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client  on partition [jmx,3] failed due to
> Partition [jmx,3] doesn't exist on 2 (kafka.server.KafkaApis)
> [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce request with
> correlation id 622369865 from client  on partition [FunnelProto,5] failed
> due to Partition [FunnelProto,5] doesn't exist on 2
> (kafka.server.KafkaApis)
>
>
> When I ran listTopics command for one of above topic, all partitions are
> under replicated (we have replication factor set to 3). Any clues on what
> could be issue and how can we get it back to working?
>
> Thanks,
> Raja.
>
>
>
> On Fri, Sep 13, 2013 at 6:26 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Ah ok. Thanks for sharing that.
> >
> >
> >
> > On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > We have 3 zookeeper node in the cluster with a hardware load balancer .
> >  In
> > > one of the zookeeper, we did not configure ensemble correctly (server.n
> > > property in zoo.cfg) . So it ended up as like 2 nodes in one cluster,
> one
> > > node in other cluster. The load balancer is randomly hitting one of 2
> > > zookeepers in two different cluster.
> > >
> > > Thanks,
> > > Raja.
> > >
> > >
> > > On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Just curious to know, what was the misconfiguration?
> > > >
> > > >
> > > > On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango
> > > > <[EMAIL PROTECTED]>wrote:
> > > >
> > > > > Thanks Neha and Jun, It turned out to be miss configuration in our
> > > > > zookeeper cluster. After correcting it everything looks good.
> > > > >
> > > > > Thanks,
> > > > > Raja.
> > > > >
> > > > >
> > > > > On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > > > Any error in the controller and the state-change log? Are brokers
> > > 2,3,4
> > > > > > alive?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango <
> > > > > [EMAIL PROTECTED]
> > > > > > >wrote:
> > > > > >
> > > > > > > We are seeing a problem that we we try to send messages to new
> > > topic
> > > > it
> > > > > > > fails kafka.common.LeaderNotAvailableException. But usually
> this
> > > > > problem
> > > > > > > will be transient and if we re-send messages to same topic will
> > > work.
> > > > > But
> > > > > > > now we tried rending message to same topic several time, but

 
+
Rajasekar Elango 2013-09-18, 17:31
+
Neha Narkhede 2013-09-18, 21:21