Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Failover for Zookeeper and Kafka


Copy link to this message
-
Re: Failover for Zookeeper and Kafka
For asynchronous mirroring, you can take a look at
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+%28MirrorMaker%29

Thanks,

Jun
On Mon, Sep 9, 2013 at 11:30 PM, Xuyen On <[EMAIL PROTECTED]> wrote:

> Hi Joe,
>
> Yes I think we'd need to setup 3 zookeeper nodes in order to support a
> single node zookeeper failure. Was just wondering what other people have
> done in their production system. Is it worth it to setup 3 zookeeper nodes
> for a 3 - 5 node kafka cluster or would a single zookeeper node be enough?
>
> On my second question we are using Kafka 0.7 right so we do not have the
> replication feature yet. I have read about the mirroring feature you
> mentioned but I am hoping someone can help me find a way to setup an
> active-active 0.7 Kafka set of clusters (one main, one hot backup) so that
> when the main cluster goes down, the producers and consumers will
> automatically switch over to the backup Kafka cluster. I've read somewhere
> that LinkedIn has an active-active setup and was wondering how they are
> able to do this if possible with 0.7 Kafka.
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Joe Stein [mailto:[EMAIL PROTECTED]]
> Sent: Monday, September 09, 2013 5:56 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Failover for Zookeeper and Kafka
>
> You want to setup a Zookeeper ensemble ( always an odd number of servers,
> three is often acceptable )
> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup
>
> And use Kafka 0.8 replication
> http://kafka.apache.org/documentation.html#replication in addition if you
> want to mirror the cluster like in another data center you could also use
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)
>
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop
> ********************************************/
>
>
> On Sep 9, 2013, at 8:39 PM, Xuyen On <[EMAIL PROTECTED]> wrote:
>
> > Hi everyone,
> >
> > I am trying to setup a Kafka cluster and have a couple of questions
> about failover.
> >
> > Has anyone deployed more than one zookeeper for a single Kafka cluster
> and have high availability so if one zookeeper node goes down, the cluster
> automatically fails over to a backup zookeeper node? If so, how is this
> done?
> >
> > My second question is how can I set up for automatic failover if I have
> a mirror secondary Kafka cluster. So, if the main Kafka cluster goes down,
> what do I need to do in order for my producers and consumers to
> automatically fail over to the backup mirror Kafka cluster. Do I need to
> code this into my producers and consumers, should I setup a DNS redirect in
> case the main Kafka cluster goes down to point to the mirror cluster, or is
> there some other configuration that I can do?
> >
> > Thanks,
> >
> > Xuyen
> >
> >
>
>