Hi,

>From docs, its not clear whether async mode and compression are related or
independent. Can you please elaborate?

One more thing - compression definitely enhances performance, but is it
true for async mode as well (assuming they are not related)?

Thanks,
Navneet Sharma

Hi Navneet,

>From docs, its not clear whether async mode and compression are related or
> independent. Can you please elaborate?
>

They are related in the sense that the DefaultEventHandler that is used in
async mode compresses a batch of messages when they are serialized.
However, compression is not tied to async mode. Sync mode can also use
compression, but it does not do batching.
>
> One more thing - compression definitely enhances performance, but is it
> true for async mode as well (assuming they are not related)?
>

Async mode basically enables batching of messages so its throughput gain is
orthogonal to that gained by compression.

Thanks,

Joel

Also, compression ratios are usually better on larger payloads, so
compression should, in most cases, be more effective when combined with
async because the batching of messages results in larger payloads, and thus
better compression ratios.

--
Felix

On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> Hi Navneet,
>
> From docs, its not clear whether async mode and compression are related or
> > independent. Can you please elaborate?
> >
>
> They are related in the sense that the DefaultEventHandler that is used in
> async mode compresses a batch of messages when they are serialized.
> However, compression is not tied to async mode. Sync mode can also use
> compression, but it does not do batching.
>
>
> >
> > One more thing - compression definitely enhances performance, but is it
> > true for async mode as well (assuming they are not related)?
> >
>
> Async mode basically enables batching of messages so its throughput gain is
> orthogonal to that gained by compression.
>
> Thanks,
>
> Joel
>

another factor to be aware of is the effect of partitions. Even with async
producer in my use case, I saw lower compression ratio because the messages
are randomly split across 5 partitions.. A simple work around was to use a
key such that each of 1000 consecutive messages go to one partition.

we are really happy with how well Kafka 'just worked' in our prototype at
twitter. will likely use it production soon..

On Wed, Apr 25, 2012 at 10:47 AM, Felix GV <[EMAIL PROTECTED]> wrote:

> Also, compression ratios are usually better on larger payloads, so
> compression should, in most cases, be more effective when combined with
> async because the batching of messages results in larger payloads, and thus
> better compression ratios.
>
> --
> Felix
>
>
>
> On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > Hi Navneet,
> >
> > From docs, its not clear whether async mode and compression are related
> or
> > > independent. Can you please elaborate?
> > >
> >
> > They are related in the sense that the DefaultEventHandler that is used
> in
> > async mode compresses a batch of messages when they are serialized.
> > However, compression is not tied to async mode. Sync mode can also use
> > compression, but it does not do batching.
> >
> >
> > >
> > > One more thing - compression definitely enhances performance, but is it
> > > true for async mode as well (assuming they are not related)?
> > >
> >
> > Async mode basically enables batching of messages so its throughput gain
> is
> > orthogonal to that gained by compression.
> >
> > Thanks,
> >
> > Joel
> >
>

Raghu,

It's good to hear that Twitter will start to use Kafka in prod soon. Once
that happens, do you think you can add your usage in the powerby wiki page?

Thanks,

Jun

On Wed, Apr 25, 2012 at 6:31 PM, Raghu Angadi <[EMAIL PROTECTED]> wrote:

> another factor to be aware of is the effect of partitions. Even with async
> producer in my use case, I saw lower compression ratio because the messages
> are randomly split across 5 partitions.. A simple work around was to use a
> key such that each of 1000 consecutive messages go to one partition.
>
> we are really happy with how well Kafka 'just worked' in our prototype at
> twitter. will likely use it production soon..
>
> On Wed, Apr 25, 2012 at 10:47 AM, Felix GV <[EMAIL PROTECTED]> wrote:
>
> > Also, compression ratios are usually better on larger payloads, so
> > compression should, in most cases, be more effective when combined with
> > async because the batching of messages results in larger payloads, and
> thus
> > better compression ratios.
> >
> > --
> > Felix
> >
> >
> >
> > On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> >
> > > Hi Navneet,
> > >
> > > From docs, its not clear whether async mode and compression are related
> > or
> > > > independent. Can you please elaborate?
> > > >
> > >
> > > They are related in the sense that the DefaultEventHandler that is used
> > in
> > > async mode compresses a batch of messages when they are serialized.
> > > However, compression is not tied to async mode. Sync mode can also use
> > > compression, but it does not do batching.
> > >
> > >
> > > >
> > > > One more thing - compression definitely enhances performance, but is
> it
> > > > true for async mode as well (assuming they are not related)?
> > > >
> > >
> > > Async mode basically enables batching of messages so its throughput
> gain
> > is
> > > orthogonal to that gained by compression.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> >
>

will surely do that. Initially this will complement existing scribe
aggregators, and later might replace the aggregators.

thanks,
Raghu.

On Wed, Apr 25, 2012 at 7:08 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Raghu,
>
> It's good to hear that Twitter will start to use Kafka in prod soon. Once
> that happens, do you think you can add your usage in the powerby wiki page?
>
> Thanks,
>
> Jun
>
> On Wed, Apr 25, 2012 at 6:31 PM, Raghu Angadi <[EMAIL PROTECTED]>
> wrote:
>
> > another factor to be aware of is the effect of partitions. Even with
> async
> > producer in my use case, I saw lower compression ratio because the
> messages
> > are randomly split across 5 partitions.. A simple work around was to use
> a
> > key such that each of 1000 consecutive messages go to one partition.
> >
> > we are really happy with how well Kafka 'just worked' in our prototype at
> > twitter. will likely use it production soon..
> >
> > On Wed, Apr 25, 2012 at 10:47 AM, Felix GV <[EMAIL PROTECTED]> wrote:
> >
> > > Also, compression ratios are usually better on larger payloads, so
> > > compression should, in most cases, be more effective when combined with
> > > async because the batching of messages results in larger payloads, and
> > thus
> > > better compression ratios.
> > >
> > > --
> > > Felix
> > >
> > >
> > >
> > > On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Hi Navneet,
> > > >
> > > > From docs, its not clear whether async mode and compression are
> related
> > > or
> > > > > independent. Can you please elaborate?
> > > > >
> > > >
> > > > They are related in the sense that the DefaultEventHandler that is
> used
> > > in
> > > > async mode compresses a batch of messages when they are serialized.
> > > > However, compression is not tied to async mode. Sync mode can also
> use
> > > > compression, but it does not do batching.
> > > >
> > > >
> > > > >
> > > > > One more thing - compression definitely enhances performance, but
> is
> > it
> > > > > true for async mode as well (assuming they are not related)?
> > > > >
> > > >
> > > > Async mode basically enables batching of messages so its throughput
> > gain
> > > is
> > > > orthogonal to that gained by compression.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > >
> >
>