Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Fetch request with correlation id 1171437 from client ReplicaFetcherThread-0-1 on partition [meetme,0] failed due to Leader not local for partition


+
Vadim Keylis 2013-06-28, 17:58
+
Vadim Keylis 2013-06-28, 18:09
+
David DeMaagd 2013-06-28, 18:25
+
Vadim Keylis 2013-06-28, 18:33
Copy link to this message
-
Re: Fetch request with correlation id 1171437 from client ReplicaFetcherThread-0-1 on partition [meetme,0] failed due to Leader not local for partition
Unless I'm misreading something, that is controlled by the
topic.metadata.refresh.interval.ms variable (defaults to 10 minutes),
and I've not seen it run longer than that (unless there was other
problems besides that going on).  

I would check the JMX values for things under "kafka.server":type="ReplicaManager",
particularly UnderReplicatedPartitions and possibly the ISR
Expand/Shrinks values - those could indicate a problem on the brokers
that is preventing things from settling down completely.  Might also
look and see if you are doing any heavy GCs (which can cause zookeeper
connection issues, which would then complicate the ISR election stuff).

--
Dave DeMaagd
[EMAIL PROTECTED] | 818 262 7958

([EMAIL PROTECTED] - Fri, Jun 28, 2013 at 11:32:42AM -0700)
> David. What is the expected time frame for the exception to continue? Its
> an hour has passed since short downtime and I still see the exception in
> kafka service logs.
>
> Thanks,
> Vadim
>
>
> On Fri, Jun 28, 2013 at 11:25 AM, David DeMaagd <[EMAIL PROTECTED]>wrote:
>
> > Getting kafka.common.NotLeaderForPartitionException for a time after a
> > node is brought back on line (especially if it is a short downtime) is
> > normal - that is because the consumers have not yet completely picked up
> > the new leader information.  If should settle shortly.
> >
> > --
> > Dave DeMaagd
> > [EMAIL PROTECTED] | 818 262 7958
> >
> > ([EMAIL PROTECTED] - Fri, Jun 28, 2013 at 11:08:46AM -0700)
> > > I want to clarify that I restarted only one kafka node, all others were
> > > running and did not require restart
> > >
> > >
> > > On Fri, Jun 28, 2013 at 10:57 AM, Vadim Keylis <[EMAIL PROTECTED]
> > >wrote:
> > >
> > > > Good morning. I have a cluster of 3 kafka nodes. They were both
> > running at
> > > > the time. I need it to make configuration change in the property file
> > and
> > > > restart kafka. I have not broker shutdown tool, but simple used pkill
> > -TERM
> > > > -u ${KAFKA_USER} -f kafka.Kafka. That suddenly cause the  exception.
> > How to
> > > > avoid this issue in the future? What's the right way to shutdown kafka
> > to
> > > > prevent Not Leder Exception
> > > >
> > > > Thanks so much in advance,
> > > > Vadim
> > > >
> > > >
> > > >
> > > > [2013-06-28 10:46:53,281] WARN [KafkaApi-1] Fetch request with
> > correlation
> > > > id 1171435 from client ReplicaFetcherThread-0-1 on partition [meetme,0]
> > > > failed due to Leader not local for partition [meetme,0] on broker 1
> > > > (kafka.server.KafkaApis)
> > > > [2013-06-28 10:46:53,282] WARN [KafkaApi-1] Fetch request with
> > correlation
> > > > id 1171436 from client ReplicaFetcherThread-0-1 on partition [meetme,0]
> > > > failed due to Leader not local for partition [meetme,0] on broker 1
> > > > (kafka.server.KafkaApis)
> > > > [2013-06-28 10:46:53,448] WARN [ReplicaFetcherThread-0-2], error for
> > > > partition [meetme,0] to broker 2 (kafka.server.ReplicaFetcherThread)
> > > > kafka.common.NotLeaderForPartitionException
> > > >         at
> > sun.reflect.GeneratedConstructorAccessor2.newInstance(Unknown
> > > > Source)
> > > >         at
> > > >
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > > >         at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > >         at java.lang.Class.newInstance0(Class.java:355)
> > > >         at java.lang.Class.newInstance(Class.java:308)
> > > >         at
> > kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> > > >         at
> > > >
> > kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:157)
> > > >         at
> > > >
> > kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:157)
> > > >         at kafka.utils.Logging$class.warn(Logging.scala:88)
> > > >         at
> > kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)

 
+
Joel Koshy 2013-06-28, 19:27
+
Vadim Keylis 2013-06-28, 20:40
+
Joel Koshy 2013-06-28, 21:28
+
Jun Rao 2013-06-29, 04:26
+
Vadim Keylis 2013-06-29, 06:27
+
Jun Rao 2013-06-29, 15:46
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB