Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Failover for Zookeeper and Kafka


Copy link to this message
-
Re: Failover for Zookeeper and Kafka
For asynchronous mirroring, you can take a look at
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+%28MirrorMaker%29

Thanks,

Jun
On Mon, Sep 9, 2013 at 11:30 PM, Xuyen On <[EMAIL PROTECTED]> wrote:

> Hi Joe,
>
> Yes I think we'd need to setup 3 zookeeper nodes in order to support a
> single node zookeeper failure. Was just wondering what other people have
> done in their production system. Is it worth it to setup 3 zookeeper nodes
> for a 3 - 5 node kafka cluster or would a single zookeeper node be enough?
>
> On my second question we are using Kafka 0.7 right so we do not have the
> replication feature yet. I have read about the mirroring feature you
> mentioned but I am hoping someone can help me find a way to setup an
> active-active 0.7 Kafka set of clusters (one main, one hot backup) so that
> when the main cluster goes down, the producers and consumers will
> automatically switch over to the backup Kafka cluster. I've read somewhere
> that LinkedIn has an active-active setup and was wondering how they are
> able to do this if possible with 0.7 Kafka.
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Joe Stein [mailto:[EMAIL PROTECTED]]
> Sent: Monday, September 09, 2013 5:56 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Failover for Zookeeper and Kafka
>
> You want to setup a Zookeeper ensemble ( always an odd number of servers,
> three is often acceptable )
> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup
>
> And use Kafka 0.8 replication
> http://kafka.apache.org/documentation.html#replication in addition if you
> want to mirror the cluster like in another data center you could also use
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)
>
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop
> ********************************************/
>
>
> On Sep 9, 2013, at 8:39 PM, Xuyen On <[EMAIL PROTECTED]> wrote:
>
> > Hi everyone,
> >
> > I am trying to setup a Kafka cluster and have a couple of questions
> about failover.
> >
> > Has anyone deployed more than one zookeeper for a single Kafka cluster
> and have high availability so if one zookeeper node goes down, the cluster
> automatically fails over to a backup zookeeper node? If so, how is this
> done?
> >
> > My second question is how can I set up for automatic failover if I have
> a mirror secondary Kafka cluster. So, if the main Kafka cluster goes down,
> what do I need to do in order for my producers and consumers to
> automatically fail over to the backup mirror Kafka cluster. Do I need to
> code this into my producers and consumers, should I setup a DNS redirect in
> case the main Kafka cluster goes down to point to the mirror cluster, or is
> there some other configuration that I can do?
> >
> > Thanks,
> >
> > Xuyen
> >
> >
>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB