Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 15565: PIG-3567 LogicalPlanPrinter throws OOM for large scripts


Copy link to this message
-
Re: Review Request 15565: PIG-3567 LogicalPlanPrinter throws OOM for large scripts
Aniket Mokashi 2013-11-23, 02:22


> On Nov. 15, 2013, 9:53 p.m., Daniel Dai wrote:
> > Didn't go through line by line, but general approach looks good. You can add a unit test to check the memory footprint.
>
> Aniket Mokashi wrote:
>     We had few scripts failing due to OOM because of this issue. With this change, they work now. I will add a test to show plan print before the change and after a change is same.

I tested this change against a few production scripts and I see the plan printing properly. It's hard to come with a backward compatibility test. I will commit this if there are no objections.
- Aniket
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15565/#review28990
-----------------------------------------------------------
On Nov. 15, 2013, 5:25 a.m., Aniket Mokashi wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15565/
> -----------------------------------------------------------
>
> (Updated Nov. 15, 2013, 5:25 a.m.)
>
>
> Review request for pig, Cheolsoo Park, Daniel Dai, Julien Le Dem, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3567
>     https://issues.apache.org/jira/browse/PIG-3567
>
>
> Repository: pig
>
>
> Description
> -------
>
> Changed LPPrinter recursion to directly write to stream so that memory footprint is reduced.
>
>
> Diffs
> -----
>
>   trunk/src/org/apache/pig/newplan/logical/optimizer/LogicalPlanPrinter.java 1538708
>
> Diff: https://reviews.apache.org/r/15565/diff/
>
>
> Testing
> -------
>
> Diff'ed explain plan of few existing scripts. No diffs found. I will try to submit a testcase to show that LP print of before after is same (test need not be committed).
>
>
> Thanks,
>
> Aniket Mokashi
>
>