Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Shallow iteration and MirrorMaker


Copy link to this message
-
Re: Shallow iteration and MirrorMaker
Thanks for your response, here is the JIRA:
https://issues.apache.org/jira/browse/KAFKA-732
On Thu, Jan 24, 2013 at 1:16 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> We haven't tried to run this. Please can you file a JIRA ?
>
>
> On Wed, Jan 23, 2013 at 9:30 AM, Maxime Brugidou
> <[EMAIL PROTECTED]>wrote:
>
> > MirrorMaker between two 0.8 clusters.
> >
> >
> > On Wed, Jan 23, 2013 at 5:54 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi Maxime,
> > >
> > > Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to
> mirror
> > > data between a Kafka 0.7 and Kafka 0.8 cluster ?
> > > We have fixed the KafkaMigrationTool to always turn off shallow
> iteration
> > > since we have to decompress data in 0.7 format before sending it to a
> 0.8
> > > broker.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am working with MirrorMaker on the 0.8 branch (so I haven't tested
> > this
> > > > on 0.7 but it looks similar).
> > > >
> > > > When I enable shallow.iterator.enable=true on the consumer config,
> (and
> > > > disable compression on the producer config, as mentioned in the wiki)
> > the
> > > > throughput is really good but the data produced is unreadable since
> the
> > > > messages are taken "as is" by the producer and i just get a bunch of
> > > bytes
> > > > stored with the ByteBufferMessageSet header and all the info.
> > > >
> > > > I tested using the kafka-console-consumer to read my mirrored data
> and
> > > it's
> > > > unreadable (unless I pipe the data to gunzip, which manages to read
> > it).
> > > >
> > > > Did I miss something? Is there anything i can activate on the
> producer
> > > side
> > > > of MirrorMaker to send the shallow message "as is"? Did anyone manage
> > to
> > > > make MirrorMaker work with shallow.iterator.enable=true?
> > > >
> > > > Thanks for your help,
> > > > Maxime
> > > >
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB