Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Unable to disable compression of output


Copy link to this message
-
Re: Unable to disable compression of output
Turns out I needed to set the option inside my pig script, as in:

SET mapred.output.compress false;

On Mon, Jul 30, 2012 at 2:21 PM, James Kebinger <[EMAIL PROTECTED]> wrote:

> Hello, I'm running a pretty simple pig job but despite my best efforts to
> disable compression, the output parts are written in in .lzo_deflate form
> like
>
> "/user/jkebinger/pigresults3.gz/part-r-00000.lzo_deflate"
>
> I'm launching the job like this JAVA_OPTS=-Dmapred.output.compress=false
> PIG_OPTS=-Dmapred.output.compress=false ../pig-0.10.0/bin/pig test.pig
>
> in a vain effort to turn that off.
>
> Can someone please point me in the right direction?
>
>
> The store statement looks like this:
>
> STORE bothcountedp into '/user/jkebinger/pigresults3.gz' USING
> PigStorage();
>
> (the filename there has varied from no extension to .gz with no effect on
> how the files inside it are treated)
>
> thanks!
>
> -James
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB