Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> 0.8 best practices for migrating / electing leaders in failure situations?


Copy link to this message
-
Re: 0.8 best practices for migrating / electing leaders in failure situations?
* Scenario 1:  BrokerID 1,2,3   Broker 2 dies.

Here, you can use reassign partitions tool and for all partitions that
had a replica on broker 2, move it to broker 4

* Scenario 2: BrokerID 1,2,3 Catastrophic failure 1,2,3 die but ZK still
there.

There is no way to recover any data here since there is nothing
available to consume data from.

Thanks,
Neha

On Fri, Mar 22, 2013 at 10:46 AM, Scott Clasen <[EMAIL PROTECTED]> wrote:
> What would the recommended practice be for the following scenarios?
>
> Running on EC2, ephemperal disks only for kafka.
>
> There are 3 kafka servers. The broker ids are always increasing. If a
> broker dies its never coming back.
>
> All topics have a replication factor of 3.
>
> * Scenario 1:  BrokerID 1,2,3   Broker 2 dies.
>
> Recover by:
>
> Boot another: BrokerID 4
> ?? run bin/kafka-reassign-partitions.sh   for any topic+partition and
> replace brokerid 2 with brokerid 4
> ?? anything else to do to cause messages to be replicated to 4??
>
> NOTE: This appears to work but not positive 4 got messages replicated to it.
>
> * Scenario 2: BrokerID 1,2,3 Catastrophic failure 1,2,3 die but ZK still
> there.
>
> Messages obviously lost.
> Recover to a functional state by:
>
> Boot 3 more: 4,5 6
> ?? run bin/kafka-reassign-partitions.sh  for all topics/partitions, swap
> 1,2,3 for 4,5,6?
> ?? rin bin/kafka-preferred-replica-election.sh for all topics/partitions
> ?? anything else to do to allow producers to start sending successfully??
>
>
> NOTE: I had some trouble with scenario 2. Will try to reproduce and open a
> ticket, if in fact my procedures for scenario 2 are correct, and I still
> cant get to a good state.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB