Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Shallow iteration and MirrorMaker


Copy link to this message
-
Re: Shallow iteration and MirrorMaker
Thanks, I will take a look

-Neha
On Thu, Jan 24, 2013 at 12:55 AM, Maxime Brugidou <[EMAIL PROTECTED]
> wrote:

> Thanks for your response, here is the JIRA:
> https://issues.apache.org/jira/browse/KAFKA-732
>
>
> On Thu, Jan 24, 2013 at 1:16 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > We haven't tried to run this. Please can you file a JIRA ?
> >
> >
> > On Wed, Jan 23, 2013 at 9:30 AM, Maxime Brugidou
> > <[EMAIL PROTECTED]>wrote:
> >
> > > MirrorMaker between two 0.8 clusters.
> > >
> > >
> > > On Wed, Jan 23, 2013 at 5:54 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hi Maxime,
> > > >
> > > > Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to
> > mirror
> > > > data between a Kafka 0.7 and Kafka 0.8 cluster ?
> > > > We have fixed the KafkaMigrationTool to always turn off shallow
> > iteration
> > > > since we have to decompress data in 0.7 format before sending it to a
> > 0.8
> > > > broker.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
> > > > <[EMAIL PROTECTED]>wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am working with MirrorMaker on the 0.8 branch (so I haven't
> tested
> > > this
> > > > > on 0.7 but it looks similar).
> > > > >
> > > > > When I enable shallow.iterator.enable=true on the consumer config,
> > (and
> > > > > disable compression on the producer config, as mentioned in the
> wiki)
> > > the
> > > > > throughput is really good but the data produced is unreadable since
> > the
> > > > > messages are taken "as is" by the producer and i just get a bunch
> of
> > > > bytes
> > > > > stored with the ByteBufferMessageSet header and all the info.
> > > > >
> > > > > I tested using the kafka-console-consumer to read my mirrored data
> > and
> > > > it's
> > > > > unreadable (unless I pipe the data to gunzip, which manages to read
> > > it).
> > > > >
> > > > > Did I miss something? Is there anything i can activate on the
> > producer
> > > > side
> > > > > of MirrorMaker to send the shallow message "as is"? Did anyone
> manage
> > > to
> > > > > make MirrorMaker work with shallow.iterator.enable=true?
> > > > >
> > > > > Thanks for your help,
> > > > > Maxime
> > > > >
> > > >
> > >
> >
>