Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka mirroring fault tolerance

Riju Kallivalappil 2013-03-22, 19:03
Copy link to this message
Re: Kafka mirroring fault tolerance
Yes, this is true if you have only 1 broker in the target cluster. If you
set up multiple brokers in the target cluster, mirror maker will send
messages to available brokers.



On Fri, Mar 22, 2013 at 12:02 PM, Riju Kallivalappil <

> Hi,
>  I've a question about fault tolerance of Kafka mirror maker (0.7.1)
> Let's say that I've a mirroring setup with topics in broker B1 mirrored to
> broker B2.
> On B2, I've Kafka mirror maker and Kafka broker process running. Now,
> following is what I noticed when the Kafka broker process on B2 is
> restarted.
> The embedded consumer in the Mirror Maker continues to consume messages
> from B1 and updates the ZK offsets even after the Kafka process is stopped.
> Pretty much all of the messages that are consumed during the time that the
> local broker process is down is lost.
> Is this expected behavior? Looking through the code, I suppose it is.
> Are there any suggestions on how to avoid this kind of data loss?
> Thanks,
> Riju

Riju Kallivalappil 2013-03-25, 18:15