Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> relation between async mode and compression


Copy link to this message
-
Re: relation between async mode and compression
will surely do that. Initially this will complement existing scribe
aggregators, and later might replace the aggregators.

thanks,
Raghu.

On Wed, Apr 25, 2012 at 7:08 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Raghu,
>
> It's good to hear that Twitter will start to use Kafka in prod soon. Once
> that happens, do you think you can add your usage in the powerby wiki page?
>
> Thanks,
>
> Jun
>
> On Wed, Apr 25, 2012 at 6:31 PM, Raghu Angadi <[EMAIL PROTECTED]>
> wrote:
>
> > another factor to be aware of is the effect of partitions. Even with
> async
> > producer in my use case, I saw lower compression ratio because the
> messages
> > are randomly split across 5 partitions.. A simple work around was to use
> a
> > key such that each of 1000 consecutive messages go to one partition.
> >
> > we are really happy with how well Kafka 'just worked' in our prototype at
> > twitter. will likely use it production soon..
> >
> > On Wed, Apr 25, 2012 at 10:47 AM, Felix GV <[EMAIL PROTECTED]> wrote:
> >
> > > Also, compression ratios are usually better on larger payloads, so
> > > compression should, in most cases, be more effective when combined with
> > > async because the batching of messages results in larger payloads, and
> > thus
> > > better compression ratios.
> > >
> > > --
> > > Felix
> > >
> > >
> > >
> > > On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Hi Navneet,
> > > >
> > > > From docs, its not clear whether async mode and compression are
> related
> > > or
> > > > > independent. Can you please elaborate?
> > > > >
> > > >
> > > > They are related in the sense that the DefaultEventHandler that is
> used
> > > in
> > > > async mode compresses a batch of messages when they are serialized.
> > > > However, compression is not tied to async mode. Sync mode can also
> use
> > > > compression, but it does not do batching.
> > > >
> > > >
> > > > >
> > > > > One more thing - compression definitely enhances performance, but
> is
> > it
> > > > > true for async mode as well (assuming they are not related)?
> > > > >
> > > >
> > > > Async mode basically enables batching of messages so its throughput
> > gain
> > > is
> > > > orthogonal to that gained by compression.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > >
> >
>