Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - 0.8 best practices for migrating / electing leaders in failure situations?


Copy link to this message
-
Re: 0.8 best practices for migrating / electing leaders in failure situations?
Scott Clasen 2013-03-23, 04:23
Thanks!

 Would there be any difference if I instead  deleted all the Kafka data from zookeeper and booted 3 instances  with different broker id? clients with cached broker id lists or any other issue?

Sent from my iPhone

On Mar 22, 2013, at 9:15 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> In scenario 2, you can bring up 3 new brokers with the same broker id. You
> won't get the data back. However, new data can be published to and consumed
> from the new brokers.
>
> Thanks,
>
> Jun
>
> On Fri, Mar 22, 2013 at 2:17 PM, Scott Clasen <[EMAIL PROTECTED]> wrote:
>
>> Thanks Neha-
>>
>> To Clarify...
>>
>> *In scenario => 1 will the new broker get all messages on the other brokers
>> replicated to it?
>>
>> *In Scenario 2 => it is clear that the data is gone, but I still need
>> producers to be able to send and consumers to receive on the same topic. In
>> my testing today I was unable to do that as I kept getting errors...so if i
>> was doing the correct steps it seems there is a bug here, basically the
>> "second-cluster-topic" topic is unusable after all 3 brokers crash, and 3
>> more are booted to replace them.  Something not quite correct in zookeeper?
>>
>> Like so
>>
>> ./bin/kafka-reassign-partitions.sh --zookeeper ... --path-to-json-file
>> reassign.json
>>
>> kafka.common.LeaderNotAvailableException: Leader not available for topic
>> second-cluster-topic partition 0
>> at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:120)
>> at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:103)
>> at
>>
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>> at
>>
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>> at
>>
>> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
>> at scala.collection.immutable.List.foreach(List.scala:45)
>> at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>> at scala.collection.immutable.List.map(List.scala:45)
>> at
>>
>> kafka.admin.AdminUtils$.kafka$admin$AdminUtils$$fetchTopicMetadataFromZk(AdminUtils.scala:103)
>> at kafka.admin.AdminUtils$.fetchTopicMetadataFromZk(AdminUtils.scala:92)
>> at kafka.admin.ListTopicCommand$.showTopic(ListTopicCommand.scala:80)
>> at
>>
>> kafka.admin.ListTopicCommand$$anonfun$main$2.apply(ListTopicCommand.scala:66)
>> at
>>
>> kafka.admin.ListTopicCommand$$anonfun$main$2.apply(ListTopicCommand.scala:65)
>> at
>>
>> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
>> at scala.collection.immutable.List.foreach(List.scala:45)
>> at kafka.admin.ListTopicCommand$.main(ListTopicCommand.scala:65)
>> at kafka.admin.ListTopicCommand.main(ListTopicCommand.scala)
>> Caused by: kafka.common.LeaderNotAvailableException: No leader exists for
>> partition 0
>> at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:117)
>> ... 16 more
>> topic: second-cluster-topic
>>
>> ./bin/kafka-preferred-replica-election.sh  --zookeeper...
>> --path-to-json-file elect.json
>>
>>
>> ....[2013-03-22 10:24:20,706] INFO Created preferred replica election path
>> with { "partitions":[ { "partition":0, "topic":"first-cluster-topic" }, {
>> "partition":0, "topic":"second-cluster-topic" } ], "version":1 }
>> (kafka.admin.PreferredReplicaLeaderElectionCommand$)
>>
>> ./bin/kafka-list-topic.sh  --zookeeper ... --topic second-cluster-topic
>>
>> 2013-03-22 10:24:30,869] ERROR Error while fetching metadata for partition
>> [second-cluster-topic,0] (kafka.admin.AdminUtils$)
>> kafka.common.LeaderNotAvailableException: Leader not available for topic
>> second-cluster-topic partition 0
>> at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:120)
>> at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:103)
>> at
>>
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>> at
>>
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>> at
>>
>> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)