Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Mirror Maker


Copy link to this message
-
Mirror Maker
Hi All,

I am hoping that mirror maker is what I'm looking for. I would like to have
complete data center fail over from one kafka cluster to another. If one
data center goes down, my producers and consumers will start using the
mirrored cluster until the primary is back online. Is that something I can
do with mirror maker?

I'm thinking of using a short TTL on the DNS entries for ZooKeeper and the
brokers and have a monitor switch them to the mirrored cluster if we detect
that the primary data center has gone offline.

Some specific questions:

1) Does mirror maker replicate the consumer offsets from the source
cluster? In other words, can a consumer safely fail over from the source
cluster to the mirrored cluster?

2) I assume that when the primary comes back online, all messages published
to the mirror cluster are not back published to the primary cluster,
correct? The mirroring is one way only?

3) Same goes for consumer offsets (if they are mirrored) in the mirrored
zookeeper. A consumer will have issues going back to the primary cluster
since the offsets will be different, correct? Even if the offsets are
mirrored I'm not sure how this would work to switch back to the primary
cluster. Maybe when the primary comes back online switch the mirror maker
to make the current cluster the source and the now back online cluster the
mirror?

Any other way that any one has used to do complete data center fail over?
I've read that doing ZooKeeper cross data center is not advised, so minus
some trickery that's out. Similarly for doing a single cluster of brokers
cross data center (in response to a previous question I posted here).

Thanks.