Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - 0.8 best practices for migrating / electing leaders in failure situations?


Copy link to this message
-
Re: 0.8 best practices for migrating / electing leaders in failure situations?
Scott Clasen 2013-03-25, 16:58
Jun Thanks. To clarify, do you mean that clients will have cached broker
lists or some other data that will make them ignore the new brokers?

Like so

topic-1 replication factor 3, on broker-ids 1,2,3
all brokers 1,2,3 die, and are never coming back.
delete all kafka data in zookeeper.
boot 4,5,6, create new topic called topic-1 repl factor 3, brokers 4,5,6

clients will/will not start sending to topic-1 on 4,5,6?

On Sun, Mar 24, 2013 at 4:01 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> If you bring up 3 new brokers with different broker ids, you won't be able
> to use them on existing topics until after you have run the partition
> reassignment tool.
>
> Thanks,
>
> Jun
>
> On Fri, Mar 22, 2013 at 9:23 PM, Scott Clasen <[EMAIL PROTECTED]> wrote:
>
> > Thanks!
> >
> >  Would there be any difference if I instead  deleted all the Kafka data
> > from zookeeper and booted 3 instances  with different broker id? clients
> > with cached broker id lists or any other issue?
> >
> > Sent from my iPhone
> >
> > On Mar 22, 2013, at 9:15 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > In scenario 2, you can bring up 3 new brokers with the same broker id.
> > You
> > > won't get the data back. However, new data can be published to and
> > consumed
> > > from the new brokers.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Mar 22, 2013 at 2:17 PM, Scott Clasen <[EMAIL PROTECTED]>
> wrote:
> > >
> > >> Thanks Neha-
> > >>
> > >> To Clarify...
> > >>
> > >> *In scenario => 1 will the new broker get all messages on the other
> > brokers
> > >> replicated to it?
> > >>
> > >> *In Scenario 2 => it is clear that the data is gone, but I still need
> > >> producers to be able to send and consumers to receive on the same
> > topic. In
> > >> my testing today I was unable to do that as I kept getting errors...so
> > if i
> > >> was doing the correct steps it seems there is a bug here, basically
> the
> > >> "second-cluster-topic" topic is unusable after all 3 brokers crash,
> and
> > 3
> > >> more are booted to replace them.  Something not quite correct in
> > zookeeper?
> > >>
> > >> Like so
> > >>
> > >> ./bin/kafka-reassign-partitions.sh --zookeeper ... --path-to-json-file
> > >> reassign.json
> > >>
> > >> kafka.common.LeaderNotAvailableException: Leader not available for
> topic
> > >> second-cluster-topic partition 0
> > >> at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:120)
> > >> at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:103)
> > >> at
> > >>
> > >>
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> > >> at
> > >>
> > >>
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> > >> at
> > >>
> > >>
> >
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> > >> at scala.collection.immutable.List.foreach(List.scala:45)
> > >> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> > >> at scala.collection.immutable.List.map(List.scala:45)
> > >> at
> > >>
> > >>
> >
> kafka.admin.AdminUtils$.kafka$admin$AdminUtils$$fetchTopicMetadataFromZk(AdminUtils.scala:103)
> > >> at
> kafka.admin.AdminUtils$.fetchTopicMetadataFromZk(AdminUtils.scala:92)
> > >> at kafka.admin.ListTopicCommand$.showTopic(ListTopicCommand.scala:80)
> > >> at
> > >>
> > >>
> >
> kafka.admin.ListTopicCommand$$anonfun$main$2.apply(ListTopicCommand.scala:66)
> > >> at
> > >>
> > >>
> >
> kafka.admin.ListTopicCommand$$anonfun$main$2.apply(ListTopicCommand.scala:65)
> > >> at
> > >>
> > >>
> >
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> > >> at scala.collection.immutable.List.foreach(List.scala:45)
> > >> at kafka.admin.ListTopicCommand$.main(ListTopicCommand.scala:65)
> > >> at kafka.admin.ListTopicCommand.main(ListTopicCommand.scala)
> > >> Caused by: kafka.common.LeaderNotAvailableException: No leader exists
> > for
> > >> partition 0
> > >> at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:117)