Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Failover for Zookeeper and Kafka

Copy link to this message
Re: Failover for Zookeeper and Kafka
You want to setup a Zookeeper ensemble ( always an odd number of servers, three is often acceptable )

And use Kafka 0.8 replication http://kafka.apache.org/documentation.html#replication in addition if you want to mirror the cluster like in another data center you could also use https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)

 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 Twitter: @allthingshadoop
On Sep 9, 2013, at 8:39 PM, Xuyen On <[EMAIL PROTECTED]> wrote:

> Hi everyone,
> I am trying to setup a Kafka cluster and have a couple of questions about failover.
> Has anyone deployed more than one zookeeper for a single Kafka cluster and have high availability so if one zookeeper node goes down, the cluster automatically fails over to a backup zookeeper node? If so, how is this done?
> My second question is how can I set up for automatic failover if I have a mirror secondary Kafka cluster. So, if the main Kafka cluster goes down, what do I need to do in order for my producers and consumers to automatically fail over to the backup mirror Kafka cluster. Do I need to code this into my producers and consumers, should I setup a DNS redirect in case the main Kafka cluster goes down to point to the mirror cluster, or is there some other configuration that I can do?
> Thanks,
> Xuyen