Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Shallow iteration and MirrorMaker


+
Maxime Brugidou 2013-01-23, 14:40
Copy link to this message
-
Re: Shallow iteration and MirrorMaker
Hi Maxime,

Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror
data between a Kafka 0.7 and Kafka 0.8 cluster ?
We have fixed the KafkaMigrationTool to always turn off shallow iteration
since we have to decompress data in 0.7 format before sending it to a 0.8
broker.

Thanks,
Neha
On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
<[EMAIL PROTECTED]>wrote:

> Hi all,
>
> I am working with MirrorMaker on the 0.8 branch (so I haven't tested this
> on 0.7 but it looks similar).
>
> When I enable shallow.iterator.enable=true on the consumer config, (and
> disable compression on the producer config, as mentioned in the wiki) the
> throughput is really good but the data produced is unreadable since the
> messages are taken "as is" by the producer and i just get a bunch of bytes
> stored with the ByteBufferMessageSet header and all the info.
>
> I tested using the kafka-console-consumer to read my mirrored data and it's
> unreadable (unless I pipe the data to gunzip, which manages to read it).
>
> Did I miss something? Is there anything i can activate on the producer side
> of MirrorMaker to send the shallow message "as is"? Did anyone manage to
> make MirrorMaker work with shallow.iterator.enable=true?
>
> Thanks for your help,
> Maxime
>

 
+
Maxime Brugidou 2013-01-23, 17:30
+
Neha Narkhede 2013-01-24, 00:16
+
Maxime Brugidou 2013-01-24, 08:55
+
Neha Narkhede 2013-01-24, 16:56
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB