Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Unable to disable compression of output


Copy link to this message
-
Re: Unable to disable compression of output
Yes, whatever I name the output directory, the files inside are compressed
with the lzo codec.

On Mon, Jul 30, 2012 at 2:52 PM, souri datta <[EMAIL PROTECTED]>wrote:

> James,
>  I may not have understood your question fully,but did you try renaming the
> file w/o the '.gz' ? In hadoop/pig, if you have bz2 extn, files get
> compressed.Hence,..
>
> On Mon, Jul 30, 2012 at 11:51 PM, James Kebinger <[EMAIL PROTECTED]
> >wrote:
>
> > Hello, I'm running a pretty simple pig job but despite my best efforts to
> > disable compression, the output parts are written in in .lzo_deflate form
> > like
> >
> > "/user/jkebinger/pigresults3.gz/part-r-00000.lzo_deflate"
> >
> > I'm launching the job like this JAVA_OPTS=-Dmapred.output.compress=false
> > PIG_OPTS=-Dmapred.output.compress=false ../pig-0.10.0/bin/pig test.pig
> >
> > in a vain effort to turn that off.
> >
> > Can someone please point me in the right direction?
> >
> >
> > The store statement looks like this:
> >
> > STORE bothcountedp into '/user/jkebinger/pigresults3.gz' USING
> > PigStorage();
> >
> > (the filename there has varied from no extension to .gz with no effect on
> > how the files inside it are treated)
> >
> > thanks!
> >
> > -James
> >
>