Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka mirroring fault tolerance


+
Riju Kallivalappil 2013-03-22, 19:03
Copy link to this message
-
Re: Kafka mirroring fault tolerance
Yes, this is true if you have only 1 broker in the target cluster. If you
set up multiple brokers in the target cluster, mirror maker will send
messages to available brokers.

Thanks,

Jun

On Fri, Mar 22, 2013 at 12:02 PM, Riju Kallivalappil <
[EMAIL PROTECTED]> wrote:

> Hi,
>  I've a question about fault tolerance of Kafka mirror maker (0.7.1)
>
> Let's say that I've a mirroring setup with topics in broker B1 mirrored to
> broker B2.
>
> On B2, I've Kafka mirror maker and Kafka broker process running. Now,
> following is what I noticed when the Kafka broker process on B2 is
> restarted.
>
> The embedded consumer in the Mirror Maker continues to consume messages
> from B1 and updates the ZK offsets even after the Kafka process is stopped.
> Pretty much all of the messages that are consumed during the time that the
> local broker process is down is lost.
>
> Is this expected behavior? Looking through the code, I suppose it is.
>
> Are there any suggestions on how to avoid this kind of data loss?
>
> Thanks,
> Riju
>

 
+
Riju Kallivalappil 2013-03-25, 18:15
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB