Hi,

I have just started looking at moving from 0.7 to 0.8 and wanted to confirm
my understanding of code in the message server/broker.

In the code for 0.8, KafkaApis.appendToLocalLog calls log.append(...,
assignOffsets = true), which then calls ByteBufferMessageSet.assignOffsets.
This method seems to uncompress and then re-compress the entire set of
messages.

Is my understanding of the code correct?
Has any testing been done on the CPU consumption / performance of the
message server to determine whether this adversely impacts message
throughput under high load?

Thanks,
Ross

 

Yes, your understanding is correct. The reason we have to recompress the
messages is to assign a unique offset to messages inside a compressed
message. Some preliminary load testing shows 30% increase in CPU, but that
is using GZIP which is known to be CPU intensive. By this week, we will
know the CPU usage for a lighter compression codec like Snappy. Will post
the results on the mailing list.

Thanks,
Neha

On Monday, March 18, 2013, Ross Black wrote:

> Hi,
>
> I have just started looking at moving from 0.7 to 0.8 and wanted to confirm
> my understanding of code in the message server/broker.
>
> In the code for 0.8, KafkaApis.appendToLocalLog calls log.append(...,
> assignOffsets = true), which then calls ByteBufferMessageSet.assignOffsets.
> This method seems to uncompress and then re-compress the entire set of
> messages.
>
> Is my understanding of the code correct?
> Has any testing been done on the CPU consumption / performance of the
> message server to determine whether this adversely impacts message
> throughput under high load?
>
> Thanks,
> Ross
>

 

HI Neha,

Thanks for the info.  I will be most interested to see what your testing
shows.
Thanks,
Ross

On 19 March 2013 17:10, Neha Narkhede <[EMAIL PROTECTED]> wrote:

> Yes, your understanding is correct. The reason we have to recompress the
> messages is to assign a unique offset to messages inside a compressed
> message. Some preliminary load testing shows 30% increase in CPU, but that
> is using GZIP which is known to be CPU intensive. By this week, we will
> know the CPU usage for a lighter compression codec like Snappy. Will post
> the results on the mailing list.
>
> Thanks,
> Neha
>
> On Monday, March 18, 2013, Ross Black wrote:
>
> > Hi,
> >
> > I have just started looking at moving from 0.7 to 0.8 and wanted to
> confirm
> > my understanding of code in the message server/broker.
> >
> > In the code for 0.8, KafkaApis.appendToLocalLog calls log.append(...,
> > assignOffsets = true), which then calls
> ByteBufferMessageSet.assignOffsets.
> > This method seems to uncompress and then re-compress the entire set of
> > messages.
> >
> > Is my understanding of the code correct?
> > Has any testing been done on the CPU consumption / performance of the
> > message server to determine whether this adversely impacts message
> > throughput under high load?
> >
> > Thanks,
> > Ross
> >
>