Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Shallow iteration and MirrorMaker


Copy link to this message
-
Shallow iteration and MirrorMaker
Hi all,

I am working with MirrorMaker on the 0.8 branch (so I haven't tested this
on 0.7 but it looks similar).

When I enable shallow.iterator.enable=true on the consumer config, (and
disable compression on the producer config, as mentioned in the wiki) the
throughput is really good but the data produced is unreadable since the
messages are taken "as is" by the producer and i just get a bunch of bytes
stored with the ByteBufferMessageSet header and all the info.

I tested using the kafka-console-consumer to read my mirrored data and it's
unreadable (unless I pipe the data to gunzip, which manages to read it).

Did I miss something? Is there anything i can activate on the producer side
of MirrorMaker to send the shallow message "as is"? Did anyone manage to
make MirrorMaker work with shallow.iterator.enable=true?

Thanks for your help,
Maxime