Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Compression in kafka 0.8


+
Vadim Keylis 2013-06-05, 04:55
+
Neha Narkhede 2013-06-05, 04:57
+
Vadim Keylis 2013-06-05, 06:17
+
Jason Rosenberg 2013-06-05, 07:14
Copy link to this message
-
Re: Compression in kafka 0.8
Jun Rao 2013-06-05, 14:41
There is no code change needed in the consumer. The message format stores
the compression codec and the consumer library knows how to parse the data.

Thanks,

Jun
On Wed, Jun 5, 2013 at 12:13 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> In this case, does the consumer code need to change, account for the
> compression (or is this handled automatically in the consumer apis?).
>
> If we decide to start compressing messages to a topic, can a consumer
> seamlessly move between the transition from uncompressed to compressed?
>
> Jason
>
>
> On Tue, Jun 4, 2013 at 11:16 PM, Vadim Keylis <[EMAIL PROTECTED]>
> wrote:
>
> > Thanks so much Neha
> >
> >
> > On Tue, Jun 4, 2013 at 9:57 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > The only way of storing compressed data on the server is to have the
> > > producer send it compressed. The server sends the data in compressed
> > format
> > > to the consumers.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Tue, Jun 4, 2013 at 9:55 PM, Vadim Keylis <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Good evening. I would like to preserve logs in kafka for 2 days. Due
> to
> > > the
> > > > size of data it must be compressed otherwise it will take enormous
> > amount
> > > > of disk space. Its my understanding that the only way to store
> messages
> > > in
> > > > kafka compressed is to enable compression on the producer side, is
> that
> > > > correct? Will they be sent to consumer also in compressed format? How
> > the
> > > > consumer can check if stream been sent compressed or not? Are there
> > other
> > > > options that will allow storing data in kafka for n number of days
> > > > compressed?
> > > >
> > > > Thanks so much,
> > > > Vadim
> > > >
> > >
> >
>