Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Leader not local for partition error?


Copy link to this message
-
Re: Leader not local for partition error?
This is because we populate the key in
ReplicaManager.highWatermarkCheckpoints using the "dirs" config, but look
up the key using log.dir.getParent. So, if you have a trailing slash in the
config, they won't match. This seems a bug that we should fix. Could you
file a jira?

Thanks,

Jun
On Tue, Jul 30, 2013 at 9:18 AM, Maxime Petazzoni <[EMAIL PROTECTED]
> wrote:

> Nope, that's on a pretty standard GNU/Linux Debian system (jessie/sid)
> running a 3.9.8-1 kernel. But you were onto something. Removing the
> trailing slash is my log.dir config value made it work.
>
> I'm not sure why this would have an impact since the log directories seem
> to be correctly parsed as File objects in LogManager.scala:
>
>   val logDirs: Array[File] = config.logDirs.map(new File(_)).toArray
>
> And in both cases the server logs reports that the log for 'test-0' was
> correctly loaded, which means that log directory is also correctly inserted
> into the logs Pool[TopicAndPartition, Log].
>
> That's about as far as my Scala knowledge goes though ;-) Let me know if
> you're able to reproduce the problem when you have a trailing slash as well.
>
> Thanks,
> /Maxime
> --
> Maxime Petazzoni
> Sr. Platform Engineer
> m 408.310.0595
> www.turn.com
>
> ________________________________________
> From: Jun Rao [[EMAIL PROTECTED]]
> Sent: Monday, July 29, 2013 9:32 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Leader not local for partition error?
>
> Are you on Windows? We have seen issues like that before on Windows. You
> may have to use "/" when configuring  "log.dirs".
>
> Thanks,
>
> Jun
>
>
> On Mon, Jul 29, 2013 at 4:50 PM, Maxime Petazzoni <
> [EMAIL PROTECTED]
> > wrote:
>
> > Same issue with the 0.8 beta1 tarball. There is something interesting in
> > state-change.log though:
> >
> > [2013-07-29 16:47:26,708] TRACE Broker 0 received LeaderAndIsr request
> > correlationId 6 from controller 0 epoch 1 starting the become-leader
> > transition for partition [test,0] (state.change.logger)
> > [2013-07-29 16:47:26,736] ERROR Error on broker 0 while processing
> > LeaderAndIsr request correlationId 6 received from controller 0 epoch 1
> for
> > partition (test,0) (state.change.logger)
> > java.util.NoSuchElementException: key not found:
> > /home/maxime/opt/kafka/data/kafka
> >         at scala.collection.MapLike$class.default(MapLike.scala:223)
> >         at scala.collection.immutable.Map$Map1.default(Map.scala:93)
> >         at scala.collection.MapLike$class.apply(MapLike.scala:134)
> >         at scala.collection.immutable.Map$Map1.apply(Map.scala:93)
> >         at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:83)
> >         ...
> >
> > I have log.dir=/home/maxime/opt/kafka/data/kafka/ in server.properties.
> > That directory obviously exists after Kafka starts, and contains:
> >
> > find /home/maxime/opt/kafka/data/kafka/
> > /home/maxime/opt/kafka/data/kafka/
> > /home/maxime/opt/kafka/data/kafka/test-0
> > /home/maxime/opt/kafka/data/kafka/test-0/00000000000000000000.log
> > /home/maxime/opt/kafka/data/kafka/test-0/00000000000000000000.index
> > /home/maxime/opt/kafka/data/kafka/.lock
> >
> > Which is expected, given I have a single 'test' topic with a single
> > partition.
> >
> > Any ideas? Can you reproduce the problem on your end with a freshly
> > extracted tarball?
> >
> > Thanks,
> > /Max
> > --
> > Maxime Petazzoni
> > Sr. Platform Engineer
> > m 408.310.0595
> > www.turn.com
> >
> > ________________________________________
> > From: Jun Rao [[EMAIL PROTECTED]]
> > Sent: Sunday, July 21, 2013 9:38 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Leader not local for partition error?
> >
> > Any error/exception in state-change or controller log? Also, could you
> try
> > the 0.8 beta1 release?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Jul 15, 2013 at 1:36 PM, Maxime Petazzoni <
> > [EMAIL PROTECTED]
> > > wrote:
> >
> > > Hi all,
> > >
> > > I'm not sure if I'm doing something wrong or if I missed a step

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB