Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Failover for Zookeeper and Kafka


Copy link to this message
-
Re: Failover for Zookeeper and Kafka
You want to setup a Zookeeper ensemble ( always an odd number of servers, three is often acceptable )
http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup

And use Kafka 0.8 replication http://kafka.apache.org/documentation.html#replication in addition if you want to mirror the cluster like in another data center you could also use https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
********************************************/
On Sep 9, 2013, at 8:39 PM, Xuyen On <[EMAIL PROTECTED]> wrote:

> Hi everyone,
>
> I am trying to setup a Kafka cluster and have a couple of questions about failover.
>
> Has anyone deployed more than one zookeeper for a single Kafka cluster and have high availability so if one zookeeper node goes down, the cluster automatically fails over to a backup zookeeper node? If so, how is this done?
>
> My second question is how can I set up for automatic failover if I have a mirror secondary Kafka cluster. So, if the main Kafka cluster goes down, what do I need to do in order for my producers and consumers to automatically fail over to the backup mirror Kafka cluster. Do I need to code this into my producers and consumers, should I setup a DNS redirect in case the main Kafka cluster goes down to point to the mirror cluster, or is there some other configuration that I can do?
>
> Thanks,
>
> Xuyen
>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB