Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Shallow iteration and MirrorMaker


+
Maxime Brugidou 2013-01-23, 14:40
+
Neha Narkhede 2013-01-23, 16:55
+
Maxime Brugidou 2013-01-23, 17:30
Copy link to this message
-
Re: Shallow iteration and MirrorMaker
Neha Narkhede 2013-01-24, 00:16
We haven't tried to run this. Please can you file a JIRA ?
On Wed, Jan 23, 2013 at 9:30 AM, Maxime Brugidou
<[EMAIL PROTECTED]>wrote:

> MirrorMaker between two 0.8 clusters.
>
>
> On Wed, Jan 23, 2013 at 5:54 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Maxime,
> >
> > Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror
> > data between a Kafka 0.7 and Kafka 0.8 cluster ?
> > We have fixed the KafkaMigrationTool to always turn off shallow iteration
> > since we have to decompress data in 0.7 format before sending it to a 0.8
> > broker.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Hi all,
> > >
> > > I am working with MirrorMaker on the 0.8 branch (so I haven't tested
> this
> > > on 0.7 but it looks similar).
> > >
> > > When I enable shallow.iterator.enable=true on the consumer config, (and
> > > disable compression on the producer config, as mentioned in the wiki)
> the
> > > throughput is really good but the data produced is unreadable since the
> > > messages are taken "as is" by the producer and i just get a bunch of
> > bytes
> > > stored with the ByteBufferMessageSet header and all the info.
> > >
> > > I tested using the kafka-console-consumer to read my mirrored data and
> > it's
> > > unreadable (unless I pipe the data to gunzip, which manages to read
> it).
> > >
> > > Did I miss something? Is there anything i can activate on the producer
> > side
> > > of MirrorMaker to send the shallow message "as is"? Did anyone manage
> to
> > > make MirrorMaker work with shallow.iterator.enable=true?
> > >
> > > Thanks for your help,
> > > Maxime
> > >
> >
>

 
+
Maxime Brugidou 2013-01-24, 08:55
+
Neha Narkhede 2013-01-24, 16:56