Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Review Request: disable optimizations via pig properties


+
Travis Crawford 2013-05-09, 21:03
+
Bill Graham 2013-05-09, 22:40
+
Travis Crawford 2013-05-10, 18:26
+
Travis Crawford 2013-05-13, 20:35
+
Travis Crawford 2013-05-13, 21:18
+
Bill Graham 2013-05-13, 23:35
+
Bill Graham 2013-05-14, 00:31
+
Travis Crawford 2013-05-14, 00:00
+
Bill Graham 2013-05-14, 15:00
Copy link to this message
-
Re: Review Request: disable optimizations via pig properties
Travis Crawford 2013-05-14, 02:19


> On May 13, 2013, 11:35 p.m., Bill Graham wrote:
> > src/docs/src/documentation/content/xdocs/perf.xml, line 493
> > <https://reviews.apache.org/r/11032/diff/2/?file=290925#file290925line493>
> >
> >     Would you please specify that setting this value in both the pig properties file and the command line (or script) will be additive.
>
> Travis Crawford wrote:
>     Currently it works like this:
>    
>     (a) -optimizer_off command-line rules are always disabled.
>     (b) The "pig.optimizer.rules.disabled" property works like other properties, where setting in the script itself overwrites previously set values (from either the command-line or pig.properties).
>    
>     Disabled rules are additive in that (a) + (b) will be disabled. However, within (b) only the last specified value of pig.optimizer.rules.disabled takes effect.
>    
>     I think this makes sense for how people will want to use the feature (and I think is consistent with how other properties work).
>    
>     * Site administrators can specify default rules to disable via pig.properties.
>     * Individual scripts can override the site defaults if needed.
>     * Invokers of pig can supplement the rules to disable.
>    
>     Thoughts? If we want to be additive within (b) we'd also need a way to remove defaults set by site administrators, since the default should be a suggestion not requirement. That would easily be achieved with a "-" prefix that would remove disabled rules, but I think we've covered the common use-cases without introducing extra complexity.
>
> Bill Graham wrote:
>     > (b) The "pig.optimizer.rules.disabled" property works like other properties, where setting in the script itself overwrites previously set values (from either the command-line or pig.properties).
>    
>     This implies SET in a script will override the command line (or properties).
>    
>     > Disabled rules are additive in that (a) + (b) will be disabled. However, within (b) only the last specified value of pig.optimizer.rules.disabled takes effect.
>    
>     This implies SET in a script (or properties) would be additive with the command line.
>    
>     Can you help clarify what I think sounds like a contradiction? Just trying to understand the implemented behavior more than propose a change to it.

RE: "This implies SET in a script will override the command line (or properties)."
A: Rules disabled via the -optimizer_off command-line flag are treated separately from values disabled via the pig property.

RE: "This implies SET in a script (or properties) would be additive with the command line."
A: Correct - SET in a script is additive with rules disabled via the command-line flag.
Rules to disable are the set of rules disabled on the command line + rules disabled via the "pig.optimizer.rules.disabled" property.

Pig's code currently uses a command-line flag to disable optimization rules rather than standard pig properties. I think the ideal state would be using a single property to disable rules because properties are how pig configuration works in general. However, since there's currently a command-line flag to disable rules it seems like we should keep it (perhaps deprecating to allow removing in a future release).

The proposal in this change is to:

* preserve existing behavior by making the command-line flag continues to disable rules as it does today
* Add a new property that also lets you disable optimization rules. This is a standard pig property that can be set in all the ways one can currently set properties.

Then we add rules disabled via the command-line flag with rules disabled via the property and that's the full list of rules to disable.
- Travis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11032/#review20516
-----------------------------------------------------------
On May 13, 2013, 8:35 p.m., Travis Crawford wrote:
>
> -----------------------------------------------------------
+
Travis Crawford 2013-05-14, 03:45
+
Bill Graham 2013-05-14, 03:39
+
Julien Le Dem 2013-05-13, 23:23
+
Travis Crawford 2013-05-13, 23:50
+
Travis Crawford 2013-05-13, 23:47
+
Julien Le Dem 2013-05-14, 15:04
+
Travis Crawford 2013-05-14, 17:23
+
Bill Graham 2013-05-14, 22:06
+
Julien Le Dem 2013-05-14, 23:13
+
Travis Crawford 2013-05-16, 00:07