I am working with MirrorMaker on the 0.8 branch (so I haven't tested this on 0.7 but it looks similar).
When I enable shallow.iterator.enable=true on the consumer config, (and disable compression on the producer config, as mentioned in the wiki) the throughput is really good but the data produced is unreadable since the messages are taken "as is" by the producer and i just get a bunch of bytes stored with the ByteBufferMessageSet header and all the info.
I tested using the kafka-console-consumer to read my mirrored data and it's unreadable (unless I pipe the data to gunzip, which manages to read it).
Did I miss something? Is there anything i can activate on the producer side of MirrorMaker to send the shallow message "as is"? Did anyone manage to make MirrorMaker work with shallow.iterator.enable=true?
Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror data between a Kafka 0.7 and Kafka 0.8 cluster ? We have fixed the KafkaMigrationTool to always turn off shallow iteration since we have to decompress data in 0.7 format before sending it to a 0.8 broker.
Thanks, Neha On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou <[EMAIL PROTECTED]>wrote: