I am working with MirrorMaker on the 0.8 branch (so I haven't tested this on 0.7 but it looks similar).
When I enable shallow.iterator.enable=true on the consumer config, (and disable compression on the producer config, as mentioned in the wiki) the throughput is really good but the data produced is unreadable since the messages are taken "as is" by the producer and i just get a bunch of bytes stored with the ByteBufferMessageSet header and all the info.
I tested using the kafka-console-consumer to read my mirrored data and it's unreadable (unless I pipe the data to gunzip, which manages to read it).
Did I miss something? Is there anything i can activate on the producer side of MirrorMaker to send the shallow message "as is"? Did anyone manage to make MirrorMaker work with shallow.iterator.enable=true?
Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror data between a Kafka 0.7 and Kafka 0.8 cluster ? We have fixed the KafkaMigrationTool to always turn off shallow iteration since we have to decompress data in 0.7 format before sending it to a 0.8 broker.
Thanks, Neha On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou <[EMAIL PROTECTED]>wrote:
-Neha On Thu, Jan 24, 2013 at 12:55 AM, Maxime Brugidou <[EMAIL PROTECTED]
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext