Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> [Discussion] Any thoughts on PIG-3457?


+
Cheolsoo Park 2013-09-30, 16:51
+
Daniel Dai 2013-09-30, 17:33
+
Cheolsoo Park 2013-09-30, 17:43
Copy link to this message
-
Re: [Discussion] Any thoughts on PIG-3457?
What about the option of leaving all of the MR specific logic in the
original classes but marking those methods as deprecated and telling people
to switch to using a MR specific object that extends the original class.
 So for example:

JobStats - Reverted to being as it was before PIG-3419 but with all MR
specific logic deprecated.
MRJobStats - Would just extend JobStats.

If we did this, external software could switch their code from using
JobStats to MRJobStats at their own pace and without breaking against any
specific version of Pig.  After a few versions the MR specific logic could
be removed from JobStats and pushed into MRJobStats and it shouldn't break
anything for people that had made that change.

I'm not familiar with all of the changes in PIG-3419 so this might not work
everywhere.
On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park <[EMAIL PROTECTED]> wrote:

> To be specific, we will need to revert all the following commits in order:
>
>
> commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982
> Author: Cheolsoo Park <[EMAIL PROTECTED]>
> Date:   Fri Sep 20 22:47:29 2013 +0000
>
>     PIG-3471: Add a base abstract class for ExecutionEngine (cheolsoo)
>
>     git-svn-id:
>
> https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68
>
> commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1
> Author: Jianyong Dai <[EMAIL PROTECTED]>
> Date:   Wed Sep 18 19:09:49 2013 +0000
>
>     PIG-3457: Provide backward compatibility for PigStatsUtil and JobStats
>
>     git-svn-id:
>
> https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68
>
> commit e85cf34c92713aa697a1cda7a9c2b3db139350f7
> Author: Cheolsoo Park <[EMAIL PROTECTED]>
> Date:   Wed Sep 18 15:37:58 2013 +0000
>
>     PIG-3464: Mark ExecType and ExecutionEngine interfaces as evolving
> (cheolsoo)
>
> commit fd8b7cdf9292b305f02386d560c25298ab492a0b
> Author: Cheolsoo Park <[EMAIL PROTECTED]>
> Date:   Fri Aug 30 20:04:29 2013 +0000
>
>     PIG-3419: Pluggable Execution Engine (achalsoni81 via cheolsoo)
>
>     git-svn-id:
>
> https://svn.apache.org/repos/asf/pig/trunk@151906213f79535-47bb-0310-9956-ffa450edef68
>
>
>
>
> On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai <[EMAIL PROTECTED]>
> wrote:
>
> > Thanks Cheolsoo! My opinion is provide backward compatibility for
> PigStats
> > is a must, otherwise it could be a havoc. I imagine PigStats is widely
> used
> > by Pig users via PigRunner and PPNL interface. People use PigStats to
> > collect MR job details of the Pig job. Though PigStats is marked for
> > Evolving, this is mostly for extending PigStats, not limiting PigStats as
> > PIG-3419 did. Even if we really need to change, we need to very well
> > communicate with users over time, Pig 0.12 is not an option.
> >
> > PIG-3457 is trying to provide a backward compatibility way for PigStats,
> > but just like Cheolsoo said, it is far from ideal. I now tend to agree
> > Rohini's suggestion on PIG-3419, rollback PIG-3419, until we find a
> better
> > way. Seems PIG-3419 is a little premature. Besides the above mentioned
> > PigStats issue, I've already found 2 additional issues:
> > 1. "explain" shows unoptimized logical plan instead of optimized one
> > 2. HangingJobKiller is removed
> >
> > How does others think?
> >
> > Thanks,
> > Daniel
> >
> >
> >
> >
> > On Mon, Sep 30, 2013 at 9:51 AM, Cheolsoo Park <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi devs,
> > >
> > > PIG-3419 <https://issues.apache.org/jira/browse/PIG-3419> broke
> backward
> > > compatibility for downstream applications such as Oozie, and
> > > PIG-3457<https://issues.apache.org/jira/browse/PIG-3457> is
> > > trying to fix it.
> > >
> > > In summary, we need to keep the old MR-specific JobStats and PigStats
> for
> > > backward compatibility sadly because they're currently exposed in
> several
> > > user-facing API including PigRunner, PigServer, etc. However, this
> > defeats
> > > the purpose of PIG-3419
> > > <https://issues.apache.org/jira/browse/PIG-3419> because
Jeremy Karn / Lead Developer
MORTAR DATA / 519 277 4391 / www.mortardata.com
+
Cheolsoo Park 2013-09-30, 18:45
+
Jeremy Karn 2013-09-30, 19:40
+
Daniel Dai 2013-09-30, 19:48
+
Jeremy Karn 2013-09-30, 19:56
+
Pradeep Gollakota 2013-09-30, 20:14
+
Alan Gates 2013-09-30, 20:28
+
Rohini Palaniswamy 2013-09-30, 21:26
+
Cheolsoo Park 2013-09-30, 21:42
+
Aniket Mokashi 2013-09-30, 22:37
+
Cheolsoo Park 2013-09-30, 22:52
+
Cheolsoo Park 2013-10-01, 04:28
+
Daniel Dai 2013-10-01, 05:17