Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request: disable optimizations via pig properties


Copy link to this message
-
Re: Review Request: disable optimizations via pig properties
Travis Crawford 2013-05-10, 18:26

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11032/#review20427
-----------------------------------------------------------
Julien raised a good question asking if "set" in the script works because query parsing may not have happened yet. He's correct - I did not explicitly test that and it doesn't work. Taking a look at how to proceed. It would be ideal if individual scripts can disable optimizations.

- Travis Crawford
On May 9, 2013, 9:03 p.m., Travis Crawford wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11032/
> -----------------------------------------------------------
>
> (Updated May 9, 2013, 9:03 p.m.)
>
>
> Review request for pig, Julien Le Dem, Bill Graham, and Feng Peng.
>
>
> Description
> -------
>
> Update pig to allow disabling optimizations via pig properties. Currently optimizations must be disabled via command-line options. Pig properties can be set in pig.properties, "set" commands in scripts themselves, and command-line -D options.
>
> The use-case is, for scripts that require certain optimizations to be disabled, allowing the script itself to disable the optimization. Currently whatever runs the script needs to specially handle disabling the optimization for that specific query.
>
>
> This addresses bug PIG-3317.
>     https://issues.apache.org/jira/browse/PIG-3317
>
>
> Diffs
> -----
>
>   src/docs/src/documentation/content/xdocs/perf.xml 108ae7e
>   src/org/apache/pig/Main.java f97ed9f
>   src/org/apache/pig/PigConstants.java ea77e97
>   src/org/apache/pig/newplan/logical/optimizer/LogicalPlanOptimizer.java d26f381
>
> Diff: https://reviews.apache.org/r/11032/diff/
>
>
> Testing
> -------
>
> Manually tested on a fully-distributed cluster.
>
> THIS FAILS:
> PIG_CONF_DIR=/etc/pig/conf ./bin/pig -c query.pig
>
> THIS WORKS:
> PIG_CONF_DIR=/etc/pig/conf ./bin/pig -Dpig.optimizer.rules.disabled=ColumnMapKeyPrune -c query.pig
>
> Notice how "-Dpig.optimizer.rules.disabled=ColumnMapKeyPrune" specifies a pig property, which could be in pig.properties, or the script itself.
>
>
> Failure message:
>
> Pig Stack Trace
> ---------------
> ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid: 97550 Input: 0 Column: 1)
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1067: Unable to explain alias null
> at org.apache.pig.PigServer.explain(PigServer.java:1057)
> at org.apache.pig.tools.grunt.GruntParser.explainCurrentBatch(GruntParser.java:419)
> at org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:351)
> at org.apache.pig.tools.grunt.Grunt.checkScript(Grunt.java:98)
> at org.apache.pig.Main.run(Main.java:607)
> at org.apache.pig.Main.main(Main.java:152)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1380)
> at org.apache.pig.PigServer.explain(PigServer.java:1042)
> ... 10 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid: 97550 Input: 0 Column: 1)
> at org.apache.pig.newplan.logical.optimizer.ProjectionPatcher$ProjectionRewriter.visit(ProjectionPatcher.java:91)