Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Leader not local for partition error?


Copy link to this message
-
Re: Leader not local for partition error?
This is because we populate the key in
ReplicaManager.highWatermarkCheckpoints using the "dirs" config, but look
up the key using log.dir.getParent. So, if you have a trailing slash in the
config, they won't match. This seems a bug that we should fix. Could you
file a jira?

Thanks,

Jun
On Tue, Jul 30, 2013 at 9:18 AM, Maxime Petazzoni <[EMAIL PROTECTED]
> wrote:

> Nope, that's on a pretty standard GNU/Linux Debian system (jessie/sid)
> running a 3.9.8-1 kernel. But you were onto something. Removing the
> trailing slash is my log.dir config value made it work.
>
> I'm not sure why this would have an impact since the log directories seem
> to be correctly parsed as File objects in LogManager.scala:
>
>   val logDirs: Array[File] = config.logDirs.map(new File(_)).toArray
>
> And in both cases the server logs reports that the log for 'test-0' was
> correctly loaded, which means that log directory is also correctly inserted
> into the logs Pool[TopicAndPartition, Log].
>
> That's about as far as my Scala knowledge goes though ;-) Let me know if
> you're able to reproduce the problem when you have a trailing slash as well.
>
> Thanks,
> /Maxime
> --
> Maxime Petazzoni
> Sr. Platform Engineer
> m 408.310.0595
> www.turn.com
>
> ________________________________________
> From: Jun Rao [[EMAIL PROTECTED]]
> Sent: Monday, July 29, 2013 9:32 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Leader not local for partition error?
>
> Are you on Windows? We have seen issues like that before on Windows. You
> may have to use "/" when configuring  "log.dirs".
>
> Thanks,
>
> Jun
>
>
> On Mon, Jul 29, 2013 at 4:50 PM, Maxime Petazzoni <
> [EMAIL PROTECTED]
> > wrote:
>
> > Same issue with the 0.8 beta1 tarball. There is something interesting in
> > state-change.log though:
> >
> > [2013-07-29 16:47:26,708] TRACE Broker 0 received LeaderAndIsr request
> > correlationId 6 from controller 0 epoch 1 starting the become-leader
> > transition for partition [test,0] (state.change.logger)
> > [2013-07-29 16:47:26,736] ERROR Error on broker 0 while processing
> > LeaderAndIsr request correlationId 6 received from controller 0 epoch 1
> for
> > partition (test,0) (state.change.logger)
> > java.util.NoSuchElementException: key not found:
> > /home/maxime/opt/kafka/data/kafka
> >         at scala.collection.MapLike$class.default(MapLike.scala:223)
> >         at scala.collection.immutable.Map$Map1.default(Map.scala:93)
> >         at scala.collection.MapLike$class.apply(MapLike.scala:134)
> >         at scala.collection.immutable.Map$Map1.apply(Map.scala:93)
> >         at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:83)
> >         ...
> >
> > I have log.dir=/home/maxime/opt/kafka/data/kafka/ in server.properties.
> > That directory obviously exists after Kafka starts, and contains:
> >
> > find /home/maxime/opt/kafka/data/kafka/
> > /home/maxime/opt/kafka/data/kafka/
> > /home/maxime/opt/kafka/data/kafka/test-0
> > /home/maxime/opt/kafka/data/kafka/test-0/00000000000000000000.log
> > /home/maxime/opt/kafka/data/kafka/test-0/00000000000000000000.index
> > /home/maxime/opt/kafka/data/kafka/.lock
> >
> > Which is expected, given I have a single 'test' topic with a single
> > partition.
> >
> > Any ideas? Can you reproduce the problem on your end with a freshly
> > extracted tarball?
> >
> > Thanks,
> > /Max
> > --
> > Maxime Petazzoni
> > Sr. Platform Engineer
> > m 408.310.0595
> > www.turn.com
> >
> > ________________________________________
> > From: Jun Rao [[EMAIL PROTECTED]]
> > Sent: Sunday, July 21, 2013 9:38 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Leader not local for partition error?
> >
> > Any error/exception in state-change or controller log? Also, could you
> try
> > the 0.8 beta1 release?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Jul 15, 2013 at 1:36 PM, Maxime Petazzoni <
> > [EMAIL PROTECTED]
> > > wrote:
> >
> > > Hi all,
> > >
> > > I'm not sure if I'm doing something wrong or if I missed a step