Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Compression in kafka 0.8


Copy link to this message
-
Re: Compression in kafka 0.8
Jason Rosenberg 2013-06-05, 07:14
In this case, does the consumer code need to change, account for the
compression (or is this handled automatically in the consumer apis?).

If we decide to start compressing messages to a topic, can a consumer
seamlessly move between the transition from uncompressed to compressed?

Jason
On Tue, Jun 4, 2013 at 11:16 PM, Vadim Keylis <[EMAIL PROTECTED]> wrote:

> Thanks so much Neha
>
>
> On Tue, Jun 4, 2013 at 9:57 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > The only way of storing compressed data on the server is to have the
> > producer send it compressed. The server sends the data in compressed
> format
> > to the consumers.
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Jun 4, 2013 at 9:55 PM, Vadim Keylis <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Good evening. I would like to preserve logs in kafka for 2 days. Due to
> > the
> > > size of data it must be compressed otherwise it will take enormous
> amount
> > > of disk space. Its my understanding that the only way to store messages
> > in
> > > kafka compressed is to enable compression on the producer side, is that
> > > correct? Will they be sent to consumer also in compressed format? How
> the
> > > consumer can check if stream been sent compressed or not? Are there
> other
> > > options that will allow storing data in kafka for n number of days
> > > compressed?
> > >
> > > Thanks so much,
> > > Vadim
> > >
> >
>