Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> MirrorMaker 0.8-Beta1 implementation issues


Copy link to this message
-
Re: MirrorMaker 0.8-Beta1 implementation issues
Our mirror maker wiki is not up to date with 0.8. In 0.8, the producer only
needs  meta.broker.list.

Thanks,

Jun
On Tue, Jul 30, 2013 at 5:23 AM, Roel van der Made <[EMAIL PROTECTED]> wrote:

> Hi al,
>
> I'm working on a DC-DC setup with two Kafka 0.8Beta1 setups and want to
> mirror the messages between two DC's (DC A and DC B) using MirrorMaker.
> The idea should not be very difficult though I encounter some, for me,
> unexplainable results and inconsistencies in the available documentation
> (some is only 0.7.x based).
>
> According to
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+%28MirrorMaker%29I should point the mirrormaker to the source cluster's ZK (DC A
> localhost:2181) and the producer to the remote (DC B) zookeeper, unless I
> use the meta.broker.list parameter, which I do. The meta.broker.list
> parameter points to the remote IP address and the three brokers running
> there. I cannot find an option on how I can point MirrorMaker to the remote
> zookeeper and besides that, the meta.broker.list parameter has shown to be
> mandatory for the producer.properties, so seems I am stuck with
> meta.broker.list.
>
> The moment I start MirrorMaker it seems in the end to only use the
> broker-list values of the local Zookeeper instance (the localhost:929[234]
> brokers) in stead of the values I put in the mirror-producer.properties
> file (shown below). I also do not see any connections ending up on the
> remote end. (remote-kafka is listed in /etc/hosts with it's public ip).
>
> So my main questions basically are;
>
> - What is the best way to define the location of the remote brokers,
> through remote ZK or through the meta.broker.list
> - Are there any other 0.8-Beta1 things i need to be aware of, or is there
> any 0.8 accurate documentation ? :)
>
>
> Thanks for any pointers, while I might well be overlooking things.
>
> Roel.
>
>
> My mirror-producer.properties file :
>
> ############################# Producer Basics #############################
>
> # list of brokers used for bootstrapping
> # format: host1:port1,host2:port2 ...
> metadata.broker.list=remote-kafka:9092,remote-kafka:9093,remote-kafka:9094
>
> # name of the partitioner class for partitioning events; default partition
> spreads data randomly
> #partitioner.class=
>
> # specifies whether the messages are sent asynchronously (async) or
> synchronously (sync)
> producer.type=async
>
> # specify the compression codec for all data generated: none , gzip,
> snappy.
> # the old config values work as well: 0, 1, 2 for none, gzip, snappy,
> respectivally
> compression.codec=none
>
> # message encoder
> serializer.class=kafka.serializer.DefaultEncoder
>
> # allow topic level compression
> #compressed.topics=
>
> ############################# Async Producer #############################
> # maximum time, in milliseconds, for buffering data on the producer queue
> queue.buffering.max.ms=11000000
>
> # the maximum size of the blocking queue for buffering on the producer
> queue.buffering.max.messages=2000000
>
> # Timeout for event enqueue:
> # 0: events will be enqueued immediately or dropped if the queue is full
> # -ve: enqueue will block indefinitely if the queue is full
> # +ve: enqueue will block up to this many milliseconds if the queue is full
> queue.enqueue.timeout.ms=-1
>
> # the number of messages batched at the producer
> #batch.num.messages=
>
>
> The log mentions:
> [playertrackingmobile is the DC A host, remote-kafka is the DC B host]
>
>
> [2013-07-30 13:40:57,437] INFO Starting mirror maker
> (kafka.tools.MirrorMaker$)
> [2013-07-30 13:40:57,596] INFO Verifying properties
> (kafka.utils.VerifiableProperties)
> [2013-07-30 13:40:57,649] INFO Property compression.codec is overridden to
> none (kafka.utils.VerifiableProperties)
> [2013-07-30 13:40:57,650] INFO Property metadata.broker.list is overridden
> to remote-kafka:9092,remote-kafka:9093,remote-kafka:9094
> (kafka.utils.VerifiableProperties)
> [2013-07-30 13:40:57,650] INFO Property producer.type is overridden to