Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Shallow iteration and MirrorMaker


Copy link to this message
-
Re: Shallow iteration and MirrorMaker
Neha Narkhede 2013-01-23, 16:55
Hi Maxime,

Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror
data between a Kafka 0.7 and Kafka 0.8 cluster ?
We have fixed the KafkaMigrationTool to always turn off shallow iteration
since we have to decompress data in 0.7 format before sending it to a 0.8
broker.

Thanks,
Neha
On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
<[EMAIL PROTECTED]>wrote:

> Hi all,
>
> I am working with MirrorMaker on the 0.8 branch (so I haven't tested this
> on 0.7 but it looks similar).
>
> When I enable shallow.iterator.enable=true on the consumer config, (and
> disable compression on the producer config, as mentioned in the wiki) the
> throughput is really good but the data produced is unreadable since the
> messages are taken "as is" by the producer and i just get a bunch of bytes
> stored with the ByteBufferMessageSet header and all the info.
>
> I tested using the kafka-console-consumer to read my mirrored data and it's
> unreadable (unless I pipe the data to gunzip, which manages to read it).
>
> Did I miss something? Is there anything i can activate on the producer side
> of MirrorMaker to send the shallow message "as is"? Did anyone manage to
> make MirrorMaker work with shallow.iterator.enable=true?
>
> Thanks for your help,
> Maxime
>