Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Snappy compression with pig


Copy link to this message
-
Re: Snappy compression with pig
Mohit Anchlia 2012-04-29, 20:41
Thanks! Is this compressing everyline or in blocks? Is it possible to set
it to compress per block?

On Sun, Apr 29, 2012 at 1:12 PM, Prashant Kommireddi <[EMAIL PROTECTED]>wrote:

> The ones you mentioned are for map output compression, not job output.
>
> On Apr 29, 2012, at 1:07 PM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
>
> > I tried these and didn't work with STORE? Is this different than the one
> > you mentioned?
> >
> > SET mapred.compress.map.output true;
> >
> > SET mapred.output.compression org.apache.hadoop.io.compress.SnappyCodec;
> >
> >
> > On Sun, Apr 29, 2012 at 11:57 AM, Prashant Kommireddi
> > <[EMAIL PROTECTED]>wrote:
> >
> >> Have you tried setting output compression to Snappy for Store?
> >>
> >> grunt> set output.compression.enabled true;
> >> grunt> set output.compression.codec
> >> org.apache.hadoop.io.compress.SnappyCodec;
> >>
> >> You should be able to read and write Snappy compressed files with
> >> PigStorage which uses Hadoop TextInputFormat internally.
> >>
> >> Thanks,
> >> Prashant
> >>
> >>
> >> On Thu, Apr 26, 2012 at 12:40 PM, Mohit Anchlia <[EMAIL PROTECTED]
> >>> wrote:
> >>
> >>> I think I need to write both store and load functions. It appears that
> >> only
> >>> intermediate output that is stored on temp location can be compressed
> >>> using:
> >>>
> >>> SET mapred.compress.map.output true;
> >>>
> >>> SET mapred.output.compression
> org.apache.hadoop.io.compress.SnappyCodec;
> >>>
> >>>
> >>>
> >>> Any pointers as to how I can store and load using snappy would be
> >> helpful.
> >>> On Thu, Apr 26, 2012 at 12:32 PM, Mohit Anchlia <
> [EMAIL PROTECTED]
> >>>> wrote:
> >>>
> >>>> I am able to write with Snappy  compression. But I don't think pig
> >>>> provides anything to read such records. Can someone suggest or point
> me
> >>> to
> >>>> relevant code that might help me write LoadFunc for it?
> >>>
> >>
>