Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> message.max.bytes and mirrormaker tool


Copy link to this message
-
Re: message.max.bytes and mirrormaker tool
You can try to reduce batch.num.messages first and see if the throughput
gets affected.

For your second question, we do not have a good solution to that since the
offsets are not consistent across data centers like you said. One way we
did is to have the consumer consuming both data centers, but keep the slave
data center's consumer do no-op; once the master data center is totally
gone, turn the nob to let the slave data center's consumers to both consume
and process. Doing so still do not give you consistency as you may consume
duplicates or data loss, but gives you a relatively close point of resuming
compared with the previous consumer.

Guozhang
On Mon, Jun 9, 2014 at 2:08 PM, Kane Kane <[EMAIL PROTECTED]> wrote: