Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> MirrorMaker 0.8-Beta1 implementation issues


Copy link to this message
-
Re: MirrorMaker 0.8-Beta1 implementation issues
Our mirror maker wiki is not up to date with 0.8. In 0.8, the producer only
needs  meta.broker.list.

Thanks,

Jun
On Tue, Jul 30, 2013 at 5:23 AM, Roel van der Made <[EMAIL PROTECTED]> wrote:

> Hi al,
>
> I'm working on a DC-DC setup with two Kafka 0.8Beta1 setups and want to
> mirror the messages between two DC's (DC A and DC B) using MirrorMaker.
> The idea should not be very difficult though I encounter some, for me,
> unexplainable results and inconsistencies in the available documentation
> (some is only 0.7.x based).
>
> According to
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+%28MirrorMaker%29I should point the mirrormaker to the source cluster's ZK (DC A
> localhost:2181) and the producer to the remote (DC B) zookeeper, unless I
> use the meta.broker.list parameter, which I do. The meta.broker.list
> parameter points to the remote IP address and the three brokers running
> there. I cannot find an option on how I can point MirrorMaker to the remote
> zookeeper and besides that, the meta.broker.list parameter has shown to be
> mandatory for the producer.properties, so seems I am stuck with
> meta.broker.list.
>
> The moment I start MirrorMaker it seems in the end to only use the
> broker-list values of the local Zookeeper instance (the localhost:929[234]
> brokers) in stead of the values I put in the mirror-producer.properties
> file (shown below). I also do not see any connections ending up on the
> remote end. (remote-kafka is listed in /etc/hosts with it's public ip).
>
> So my main questions basically are;
>
> - What is the best way to define the location of the remote brokers,
> through remote ZK or through the meta.broker.list
> - Are there any other 0.8-Beta1 things i need to be aware of, or is there
> any 0.8 accurate documentation ? :)
>
>
> Thanks for any pointers, while I might well be overlooking things.
>
> Roel.
>
>
> My mirror-producer.properties file :
>
> ############################# Producer Basics #############################
>
> # list of brokers used for bootstrapping
> # format: host1:port1,host2:port2 ...
> metadata.broker.list=remote-kafka:9092,remote-kafka:9093,remote-kafka:9094
>
> # name of the partitioner class for partitioning events; default partition
> spreads data randomly
> #partitioner.class=
>
> # specifies whether the messages are sent asynchronously (async) or
> synchronously (sync)
> producer.type=async
>
> # specify the compression codec for all data generated: none , gzip,
> snappy.
> # the old config values work as well: 0, 1, 2 for none, gzip, snappy,
> respectivally
> compression.codec=none
>
> # message encoder
> serializer.class=kafka.serializer.DefaultEncoder
>
> # allow topic level compression
> #compressed.topics=
>
> ############################# Async Producer #############################
> # maximum time, in milliseconds, for buffering data on the producer queue
> queue.buffering.max.ms=11000000
>
> # the maximum size of the blocking queue for buffering on the producer
> queue.buffering.max.messages=2000000
>
> # Timeout for event enqueue:
> # 0: events will be enqueued immediately or dropped if the queue is full
> # -ve: enqueue will block indefinitely if the queue is full
> # +ve: enqueue will block up to this many milliseconds if the queue is full
> queue.enqueue.timeout.ms=-1
>
> # the number of messages batched at the producer
> #batch.num.messages=
>
>
> The log mentions:
> [playertrackingmobile is the DC A host, remote-kafka is the DC B host]
>
>
> [2013-07-30 13:40:57,437] INFO Starting mirror maker
> (kafka.tools.MirrorMaker$)
> [2013-07-30 13:40:57,596] INFO Verifying properties
> (kafka.utils.VerifiableProperties)
> [2013-07-30 13:40:57,649] INFO Property compression.codec is overridden to
> none (kafka.utils.VerifiableProperties)
> [2013-07-30 13:40:57,650] INFO Property metadata.broker.list is overridden
> to remote-kafka:9092,remote-kafka:9093,remote-kafka:9094
> (kafka.utils.VerifiableProperties)
> [2013-07-30 13:40:57,650] INFO Property producer.type is overridden to

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB