-Re: Snappy compression with pig
Mohit Anchlia 2012-04-29, 20:06
I tried these and didn't work with STORE? Is this different than the one
SET mapred.compress.map.output true;
SET mapred.output.compression org.apache.hadoop.io.compress.SnappyCodec;
On Sun, Apr 29, 2012 at 11:57 AM, Prashant Kommireddi
> Have you tried setting output compression to Snappy for Store?
> grunt> set output.compression.enabled true;
> grunt> set output.compression.codec
> You should be able to read and write Snappy compressed files with
> PigStorage which uses Hadoop TextInputFormat internally.
> On Thu, Apr 26, 2012 at 12:40 PM, Mohit Anchlia <[EMAIL PROTECTED]
> > I think I need to write both store and load functions. It appears that
> > intermediate output that is stored on temp location can be compressed
> > using:
> > SET mapred.compress.map.output true;
> > SET mapred.output.compression org.apache.hadoop.io.compress.SnappyCodec;
> > Any pointers as to how I can store and load using snappy would be
> > On Thu, Apr 26, 2012 at 12:32 PM, Mohit Anchlia <[EMAIL PROTECTED]
> > >wrote:
> > > I am able to write with Snappy compression. But I don't think pig
> > > provides anything to read such records. Can someone suggest or point me
> > to
> > > relevant code that might help me write LoadFunc for it?