Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: failover strategy


Copy link to this message
-
Re: failover strategy
LinkedIn uses the first method for cross DC mirroring. For the second
method, there are 2 main issues. (1) Kafka depends on the ZK service to be
always available. For a ZK cluster to be available, you need a majority of
ZK servers to be up. If you set up a ZK cluster spanning only 2 data
centers, a single DC failure may make the ZK cluster unavailable. You can
set up a ZK cluster spanning 3 or more DCs, which allows you tolerate at
least 1 DC failure. (2) Long network latency across DCs. In order for the
follow to keep up with the leader in a different DC, you need to tune
parameters like replica.lag.max.messages, replica.lag.time.max.ms,
and replica.socket.receive.buffer.bytes to amortize the long network
latency.

Thanks,

Jun
On Sat, Jun 29, 2013 at 10:50 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> The first method may lose message if cluster A is permanently down or
> cannot
> restart right away as B always lags behind A. Even with mirroring, B has
> to wait
> to get missing msg until A is back. So it is not ideal. What type of
> solution did
> you use at linkedin?
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Joel Koshy [mailto:[EMAIL PROTECTED]]
> Sent: Friday, June 28, 2013 8:59 PM
> To: [EMAIL PROTECTED]
> Subject: Re: failover strategy
>
> The second method (replication across DCs) is not recommended.
> The first set up would work provided the set of topics you are mirroring
> from A->B is disjoint from the set of topics you are mirroring from B->A
> (i.e., to avoid a mirroring loop).
>
> Joel
>
> On Fri, Jun 28, 2013 at 5:29 PM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I can think of two failover strategies. I am not sure which one is the
> right way to go.
> >
> > First method. set up kafka server A on cluster 1 and set up another
> server B on cluster 2.
> > The two clusters are in different data centers. Use customized
> > mirrormaker to sync between the two servers. Use one server in
> > production and use the other one as contingency. If server A is down,
> server B will be used (this can be transparent to publishers/consumers).
> > There may be a lag between the two servers before server A is down .
> > But after A is back, the customized mirrormaker can sync the two. And
> > eventually B will have all the data A had before the failure.
> >
> > Second method. Set up one kafka server using cluster 1 and cluster 2.
> > When creating a topic , always use two replications. For each
> > partition, assign one replication to a broker in cluster 1 and assign
> > the other replication to a broker in cluster 2. So kafka will handle the
> syncing and failover for the two clusters. Is that a right (expected) way
> to use kafka?
> >
> >
> > Regards,
> >
> > Libo
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB