Hi all,

I am working with MirrorMaker on the 0.8 branch (so I haven't tested this
on 0.7 but it looks similar).

When I enable shallow.iterator.enable=true on the consumer config, (and
disable compression on the producer config, as mentioned in the wiki) the
throughput is really good but the data produced is unreadable since the
messages are taken "as is" by the producer and i just get a bunch of bytes
stored with the ByteBufferMessageSet header and all the info.

I tested using the kafka-console-consumer to read my mirrored data and it's
unreadable (unless I pipe the data to gunzip, which manages to read it).

Did I miss something? Is there anything i can activate on the producer side
of MirrorMaker to send the shallow message "as is"? Did anyone manage to
make MirrorMaker work with shallow.iterator.enable=true?

Thanks for your help,
Maxime

 

Hi Maxime,

Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror
data between a Kafka 0.7 and Kafka 0.8 cluster ?
We have fixed the KafkaMigrationTool to always turn off shallow iteration
since we have to decompress data in 0.7 format before sending it to a 0.8
broker.

Thanks,
Neha
On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
<[EMAIL PROTECTED]>wrote:

> Hi all,
>
> I am working with MirrorMaker on the 0.8 branch (so I haven't tested this
> on 0.7 but it looks similar).
>
> When I enable shallow.iterator.enable=true on the consumer config, (and
> disable compression on the producer config, as mentioned in the wiki) the
> throughput is really good but the data produced is unreadable since the
> messages are taken "as is" by the producer and i just get a bunch of bytes
> stored with the ByteBufferMessageSet header and all the info.
>
> I tested using the kafka-console-consumer to read my mirrored data and it's
> unreadable (unless I pipe the data to gunzip, which manages to read it).
>
> Did I miss something? Is there anything i can activate on the producer side
> of MirrorMaker to send the shallow message "as is"? Did anyone manage to
> make MirrorMaker work with shallow.iterator.enable=true?
>
> Thanks for your help,
> Maxime
>

 

MirrorMaker between two 0.8 clusters.
On Wed, Jan 23, 2013 at 5:54 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Hi Maxime,
>
> Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror
> data between a Kafka 0.7 and Kafka 0.8 cluster ?
> We have fixed the KafkaMigrationTool to always turn off shallow iteration
> since we have to decompress data in 0.7 format before sending it to a 0.8
> broker.
>
> Thanks,
> Neha
>
>
> On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
> <[EMAIL PROTECTED]>wrote:
>
> > Hi all,
> >
> > I am working with MirrorMaker on the 0.8 branch (so I haven't tested this
> > on 0.7 but it looks similar).
> >
> > When I enable shallow.iterator.enable=true on the consumer config, (and
> > disable compression on the producer config, as mentioned in the wiki) the
> > throughput is really good but the data produced is unreadable since the
> > messages are taken "as is" by the producer and i just get a bunch of
> bytes
> > stored with the ByteBufferMessageSet header and all the info.
> >
> > I tested using the kafka-console-consumer to read my mirrored data and
> it's
> > unreadable (unless I pipe the data to gunzip, which manages to read it).
> >
> > Did I miss something? Is there anything i can activate on the producer
> side
> > of MirrorMaker to send the shallow message "as is"? Did anyone manage to
> > make MirrorMaker work with shallow.iterator.enable=true?
> >
> > Thanks for your help,
> > Maxime
> >
>

 

We haven't tried to run this. Please can you file a JIRA ?
On Wed, Jan 23, 2013 at 9:30 AM, Maxime Brugidou
<[EMAIL PROTECTED]>wrote:

> MirrorMaker between two 0.8 clusters.
>
>
> On Wed, Jan 23, 2013 at 5:54 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Maxime,
> >
> > Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror
> > data between a Kafka 0.7 and Kafka 0.8 cluster ?
> > We have fixed the KafkaMigrationTool to always turn off shallow iteration
> > since we have to decompress data in 0.7 format before sending it to a 0.8
> > broker.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Hi all,
> > >
> > > I am working with MirrorMaker on the 0.8 branch (so I haven't tested
> this
> > > on 0.7 but it looks similar).
> > >
> > > When I enable shallow.iterator.enable=true on the consumer config, (and
> > > disable compression on the producer config, as mentioned in the wiki)
> the
> > > throughput is really good but the data produced is unreadable since the
> > > messages are taken "as is" by the producer and i just get a bunch of
> > bytes
> > > stored with the ByteBufferMessageSet header and all the info.
> > >
> > > I tested using the kafka-console-consumer to read my mirrored data and
> > it's
> > > unreadable (unless I pipe the data to gunzip, which manages to read
> it).
> > >
> > > Did I miss something? Is there anything i can activate on the producer
> > side
> > > of MirrorMaker to send the shallow message "as is"? Did anyone manage
> to
> > > make MirrorMaker work with shallow.iterator.enable=true?
> > >
> > > Thanks for your help,
> > > Maxime
> > >
> >
>

 

Thanks for your response, here is the JIRA:
https://issues.apache.org/jira/browse/KAFKA-732
On Thu, Jan 24, 2013 at 1:16 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> We haven't tried to run this. Please can you file a JIRA ?
>
>
> On Wed, Jan 23, 2013 at 9:30 AM, Maxime Brugidou
> <[EMAIL PROTECTED]>wrote:
>
> > MirrorMaker between two 0.8 clusters.
> >
> >
> > On Wed, Jan 23, 2013 at 5:54 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi Maxime,
> > >
> > > Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to
> mirror
> > > data between a Kafka 0.7 and Kafka 0.8 cluster ?
> > > We have fixed the KafkaMigrationTool to always turn off shallow
> iteration
> > > since we have to decompress data in 0.7 format before sending it to a
> 0.8
> > > broker.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am working with MirrorMaker on the 0.8 branch (so I haven't tested
> > this
> > > > on 0.7 but it looks similar).
> > > >
> > > > When I enable shallow.iterator.enable=true on the consumer config,
> (and
> > > > disable compression on the producer config, as mentioned in the wiki)
> > the
> > > > throughput is really good but the data produced is unreadable since
> the
> > > > messages are taken "as is" by the producer and i just get a bunch of
> > > bytes
> > > > stored with the ByteBufferMessageSet header and all the info.
> > > >
> > > > I tested using the kafka-console-consumer to read my mirrored data
> and
> > > it's
> > > > unreadable (unless I pipe the data to gunzip, which manages to read
> > it).
> > > >
> > > > Did I miss something? Is there anything i can activate on the
> producer
> > > side
> > > > of MirrorMaker to send the shallow message "as is"? Did anyone manage
> > to
> > > > make MirrorMaker work with shallow.iterator.enable=true?
> > > >
> > > > Thanks for your help,
> > > > Maxime
> > > >
> > >
> >
>

 

Thanks, I will take a look

-Neha
On Thu, Jan 24, 2013 at 12:55 AM, Maxime Brugidou <[EMAIL PROTECTED]
> wrote:

> Thanks for your response, here is the JIRA:
> https://issues.apache.org/jira/browse/KAFKA-732
>
>
> On Thu, Jan 24, 2013 at 1:16 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > We haven't tried to run this. Please can you file a JIRA ?
> >
> >
> > On Wed, Jan 23, 2013 at 9:30 AM, Maxime Brugidou
> > <[EMAIL PROTECTED]>wrote:
> >
> > > MirrorMaker between two 0.8 clusters.
> > >
> > >
> > > On Wed, Jan 23, 2013 at 5:54 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hi Maxime,
> > > >
> > > > Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to
> > mirror
> > > > data between a Kafka 0.7 and Kafka 0.8 cluster ?
> > > > We have fixed the KafkaMigrationTool to always turn off shallow
> > iteration
> > > > since we have to decompress data in 0.7 format before sending it to a
> > 0.8
> > > > broker.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou
> > > > <[EMAIL PROTECTED]>wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am working with MirrorMaker on the 0.8 branch (so I haven't
> tested
> > > this
> > > > > on 0.7 but it looks similar).
> > > > >
> > > > > When I enable shallow.iterator.enable=true on the consumer config,
> > (and
> > > > > disable compression on the producer config, as mentioned in the
> wiki)
> > > the
> > > > > throughput is really good but the data produced is unreadable since
> > the
> > > > > messages are taken "as is" by the producer and i just get a bunch
> of
> > > > bytes
> > > > > stored with the ByteBufferMessageSet header and all the info.
> > > > >
> > > > > I tested using the kafka-console-consumer to read my mirrored data
> > and
> > > > it's
> > > > > unreadable (unless I pipe the data to gunzip, which manages to read
> > > it).
> > > > >
> > > > > Did I miss something? Is there anything i can activate on the
> > producer
> > > > side
> > > > > of MirrorMaker to send the shallow message "as is"? Did anyone
> manage
> > > to
> > > > > make MirrorMaker work with shallow.iterator.enable=true?
> > > > >
> > > > > Thanks for your help,
> > > > > Maxime
> > > > >
> > > >
> > >
> >
>