Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - relation between async mode and compression


Copy link to this message
-
Re: relation between async mode and compression
Raghu Angadi 2012-04-26, 01:31
another factor to be aware of is the effect of partitions. Even with async
producer in my use case, I saw lower compression ratio because the messages
are randomly split across 5 partitions.. A simple work around was to use a
key such that each of 1000 consecutive messages go to one partition.

we are really happy with how well Kafka 'just worked' in our prototype at
twitter. will likely use it production soon..

On Wed, Apr 25, 2012 at 10:47 AM, Felix GV <[EMAIL PROTECTED]> wrote:

> Also, compression ratios are usually better on larger payloads, so
> compression should, in most cases, be more effective when combined with
> async because the batching of messages results in larger payloads, and thus
> better compression ratios.
>
> --
> Felix
>
>
>
> On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > Hi Navneet,
> >
> > From docs, its not clear whether async mode and compression are related
> or
> > > independent. Can you please elaborate?
> > >
> >
> > They are related in the sense that the DefaultEventHandler that is used
> in
> > async mode compresses a batch of messages when they are serialized.
> > However, compression is not tied to async mode. Sync mode can also use
> > compression, but it does not do batching.
> >
> >
> > >
> > > One more thing - compression definitely enhances performance, but is it
> > > true for async mode as well (assuming they are not related)?
> > >
> >
> > Async mode basically enables batching of messages so its throughput gain
> is
> > orthogonal to that gained by compression.
> >
> > Thanks,
> >
> > Joel
> >
>