Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.


+
Rajasekar Elango 2013-08-27, 15:54
+
Guozhang Wang 2013-08-27, 16:45
+
Neha Narkhede 2013-08-27, 16:52
+
Rajasekar Elango 2013-08-27, 19:38
Copy link to this message
-
Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.
Hello Rajasekar,

The remove fetcher log entry is normal under addition of partitions, since
they indicate that some leader changes have happened so brokers are closing
the fetchers to the old leaders.

I just realized that the console Producer does not have the
message.send.max.retries options yet. Could you file a JIRA for this and I
will followup to add this option? As for now you can hard modify the
default value from 3 to a larger number.

Guozhang
On Tue, Aug 27, 2013 at 12:37 PM, Rajasekar Elango
<[EMAIL PROTECTED]>wrote:

> Thanks Neha & Guozhang,
>
> When I ran StateChangeLogMerger, I am seeing this message repeated 16 times
> for each partition:
>
> [2013-08-27 12:30:02,535] INFO [ReplicaFetcherManager on broker 1] Removing
> fetcher for partition [test-60,13] (kafka.server.ReplicaFetcherManager)
> [2013-08-27 12:30:02,536] INFO [Log Manager on Broker 1] Created log for
> partition [test-60,13] in
> /home/relango/dev/mandm/kafka/main/target/dist/mandm-kafka/kafka-data.
> (kafka.log.LogManager)
>
> I am also seeing .log and .index files created for this topic in data dir.
> Also list topic command shows leaders, replicas and isrs for all
> partitions. Do you still think increasing num of retries would help or is
> it some other issue..? Also console Producer doesn't seem to  have option
> to set num of retries. Is there a way to configure num of retries for
> console producer ?
>
> Thanks,
> Raja.
>
>
> On Tue, Aug 27, 2013 at 12:52 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > As Guozhang said, your producer might give up sooner than the leader
> > election completes for the new topic. To confirm if your producer gave up
> > too soon, you can run the state change log merge tool for this topic and
> > see when the leader election finished for all partitions
> >
> > ./bin/kafka-run-class.sh kafka.tools.StateChangeLogMerger --logs
> <location
> > to all state change logs> --topic <topic>
> >
> > Note that this tool requires you to give the state change logs for all
> > brokers in the cluster.
> >
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Aug 27, 2013 at 9:45 AM, Guozhang Wang <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hello Rajasekar,
> > >
> > > In 0.8 producers keep a cache of the partition -> leader_broker_id map
> > > which is used to determine to which brokers should the messages be
> sent.
> > > After new partitions are added, the cache on the producer has not
> > populated
> > > yet hence it will throw this exception. The producer will then try to
> > > refresh its cache by asking the brokers "who are the leaders of these
> new
> > > partitions that I do not know of before". The brokers at the beginning
> > also
> > > do not know this information, and will only get this information from
> > > controller which will only propagation the leader information after the
> > > leader elections have all been finished.
> > >
> > > If you set num.retries to 3 then it is possible that producer gives up
> > too
> > > soon before the leader info ever propagated to producers, hence to
> > > producers also. Could you try to increase producer.num.retries and see
> if
> > > the producer can eventually succeed in re-trying?
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Aug 27, 2013 at 8:53 AM, Rajasekar Elango <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > We recently increased number of partitions from 4 to 16 and after
> that
> > > > console producer mostly fails with LeaderNotAvailableException and
> > exits
> > > > after 3 tries:
> > > >
> > > > Here is last few lines of console producer log:
> > > >
> > > > No partition metadata for topic test-41 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test-41]: class
> > > > kafka.common.LeaderNotAvailableException
> > > >  (kafka.producer.BrokerPartitionInfo)
> > > > [2013-08-27 08:29:30,271] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test-41
 
+
Rajasekar Elango 2013-08-27, 21:39
+
Guozhang Wang 2013-08-28, 00:33
+
Rajasekar Elango 2013-08-28, 15:37
+
Neha Narkhede 2013-08-28, 16:50
+
Rajasekar Elango 2013-08-28, 16:58
+
Guozhang Wang 2013-08-28, 17:11
+
Rajasekar Elango 2013-08-29, 16:16