Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - relation between async mode and compression


Copy link to this message
-
Re: relation between async mode and compression
Raghu Angadi 2012-04-26, 03:20
will surely do that. Initially this will complement existing scribe
aggregators, and later might replace the aggregators.

thanks,
Raghu.

On Wed, Apr 25, 2012 at 7:08 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Raghu,
>
> It's good to hear that Twitter will start to use Kafka in prod soon. Once
> that happens, do you think you can add your usage in the powerby wiki page?
>
> Thanks,
>
> Jun
>
> On Wed, Apr 25, 2012 at 6:31 PM, Raghu Angadi <[EMAIL PROTECTED]>
> wrote:
>
> > another factor to be aware of is the effect of partitions. Even with
> async
> > producer in my use case, I saw lower compression ratio because the
> messages
> > are randomly split across 5 partitions.. A simple work around was to use
> a
> > key such that each of 1000 consecutive messages go to one partition.
> >
> > we are really happy with how well Kafka 'just worked' in our prototype at
> > twitter. will likely use it production soon..
> >
> > On Wed, Apr 25, 2012 at 10:47 AM, Felix GV <[EMAIL PROTECTED]> wrote:
> >
> > > Also, compression ratios are usually better on larger payloads, so
> > > compression should, in most cases, be more effective when combined with
> > > async because the batching of messages results in larger payloads, and
> > thus
> > > better compression ratios.
> > >
> > > --
> > > Felix
> > >
> > >
> > >
> > > On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > Hi Navneet,
> > > >
> > > > From docs, its not clear whether async mode and compression are
> related
> > > or
> > > > > independent. Can you please elaborate?
> > > > >
> > > >
> > > > They are related in the sense that the DefaultEventHandler that is
> used
> > > in
> > > > async mode compresses a batch of messages when they are serialized.
> > > > However, compression is not tied to async mode. Sync mode can also
> use
> > > > compression, but it does not do batching.
> > > >
> > > >
> > > > >
> > > > > One more thing - compression definitely enhances performance, but
> is
> > it
> > > > > true for async mode as well (assuming they are not related)?
> > > > >
> > > >
> > > > Async mode basically enables batching of messages so its throughput
> > gain
> > > is
> > > > orthogonal to that gained by compression.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > >
> >
>