Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Unable to disable compression of output


Copy link to this message
-
Re: Unable to disable compression of output
James Kebinger 2012-07-30, 19:24
Turns out I needed to set the option inside my pig script, as in:

SET mapred.output.compress false;

On Mon, Jul 30, 2012 at 2:21 PM, James Kebinger <[EMAIL PROTECTED]> wrote:

> Hello, I'm running a pretty simple pig job but despite my best efforts to
> disable compression, the output parts are written in in .lzo_deflate form
> like
>
> "/user/jkebinger/pigresults3.gz/part-r-00000.lzo_deflate"
>
> I'm launching the job like this JAVA_OPTS=-Dmapred.output.compress=false
> PIG_OPTS=-Dmapred.output.compress=false ../pig-0.10.0/bin/pig test.pig
>
> in a vain effort to turn that off.
>
> Can someone please point me in the right direction?
>
>
> The store statement looks like this:
>
> STORE bothcountedp into '/user/jkebinger/pigresults3.gz' USING
> PigStorage();
>
> (the filename there has varied from no extension to .gz with no effect on
> how the files inside it are treated)
>
> thanks!
>
> -James
>