Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: KeeperErrorCode = BadVersion


Copy link to this message
-
Re: KeeperErrorCode = BadVersion
I see. Ideally, this error state is recoverable since we re-read the new
value from zookeeper before retrying the conditional write. Can you
describe how to reproduce this issue? Did you try deleting topics manually
that might've somehow led to this issue?

Thanks,
Neha
On Fri, Aug 23, 2013 at 10:51 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> Hi Neha,
>
> This is not a logging issue. That broker' log is flooded by this exception.
> The exception first appeared early this morning and kept flooding the log.
> The broker is not in any topic's ISR so it will not be a leader.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Friday, August 23, 2013 1:40 PM
> To: [EMAIL PROTECTED]
> Subject: Re: KeeperErrorCode = BadVersion
>
> I think we can certainly improve the logging, but this is normal when the
> leader tries to shrink/expand the ISR when the controller is in the middle
> of shrinking ISR or electing a new leader for the same partition.
>
> Could you please file a JIRA to improve the quality of logging in this
> case?
>
> Thanks,
> Neha
>
>
> On Fri, Aug 23, 2013 at 10:28 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>
> > Hi team,
> >
> > During normal operation, all of a sudden, we found many exceptions in
> > the log like this:
> >
> > It seems one thread' zookeeper's data is written unexpectedly by some
> > other thread.
> > Any expertise will be appreciated.
> >
> > [2013-08-23 13:17:00,622] INFO Partition [our.own.topic one.default,0]
> > on broker 1: Cached zkVersion [4] not equal to that in zo okeeper,
> > skip updating ISR (kafka.cluster.Partition)
> > [2013-08-23 13:17:00,622] INFO Partition [our.own.topic.two.default,2]
> > on broker 1: Shrinking ISR for partition [our.own.topic.two,2] from
> > 1,2,3 to 1 (kafka.cluster.Partition)
> > [2013-08-23 13:17:00,623] ERROR Conditional update of path /brokers/t
> > opics/our.own.topic.two/partitions/2/state with d ata {
> > "controller_epoch":81, "isr":[ 1 ], "leader":1, "leader_epoch":
> > 0, "version":1 } and expected version 10 failed (kafka.utils.ZkUtils$
> > )
> > org.I0Itec.zkclient.exception.ZkBadVersionException: org.apache.zooke
> > eper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion
> > for /brokers/topics/our.own.topic.two/partitions/2/state
> >         at
> > org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:51)
> >         at
> > org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> >         at org.I0Itec.zkclient.ZkClient.writeData(ZkClient.java:809)
> >         at
> > kafka.utils.ZkUtils$.conditionalUpdatePersistentPath(ZkUtils.scala:330)
> >         at kafka.cluster.Partition.updateIsr(Partition.scala:347)
> >         at kafka.cluster.Partition.maybeShrinkIsr(Partition.scala:291)
> >         at
> >
> kafka.server.ReplicaManager$$anonfun$kafka$server$ReplicaManager$$maybeShrinkIsr$2.apply(ReplicaManager.scala:285)
> >         at
> >
> kafka.server.ReplicaManager$$anonfun$kafka$server$ReplicaManager$$maybeShrinkIsr$2.apply(ReplicaManager.scala:285)
> >         at scala.collection.mutable.HashSet.foreach(HashSet.scala:61)
> >         at
> >
> kafka.server.ReplicaManager.kafka$server$ReplicaManager$$maybeShrinkIsr(ReplicaManager.scala:285)
> >         at
> >
> kafka.server.ReplicaManager$$anonfun$startup$1.apply$mcV$sp(ReplicaManager.scala:108)
> >         at kafka.utils.Utils$$anon$2.run(Utils.scala:67)
> >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >         at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> >         at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >         at

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB