Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> [Discussion] Any thoughts on PIG-3457?


Copy link to this message
-
Re: [Discussion] Any thoughts on PIG-3457?
Hi Jeremy,

What you're saying makes sense, and patch is welcome. ;-) But complexity
comes from that there are many classes that are associated with one
another, and it seems necessary to bring back all of them together in order
to provide full backward compatibility.

After spending many hours on the weekend, I concluded that adding more
workarounds (classes, methods, packages, etc) to the current code makes it
only less maintainable and readable. So I prefer a simpler approach.

For eg, we can just publish two jars - pig.jar w/ old API and pig-new.jar
w/ new API - maybe not in 0.12 but in 0.13. Since we already have a
tez-branch, we can use it to manage the new version of classes. Then, users
can switch to pig-new.jar gradually in 0.13 and 0.14. When we finally merge
tez-branch into trunk, we can publish a single jar again.

Of course, this is not trivial either because we have to maintain two
branches. But I feel that managing two branches independently is easier
than maintaining all sorts of workarounds for backward compatibility in the
source code. In addition, we will have more flexibility in terms of
designing new API because we will be completely free from backward
compatibility. No?

Thanks,
Cheolsoo

On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn <[EMAIL PROTECTED]> wrote:

> What about the option of leaving all of the MR specific logic in the
> original classes but marking those methods as deprecated and telling people
> to switch to using a MR specific object that extends the original class.
>  So for example:
>
> JobStats - Reverted to being as it was before PIG-3419 but with all MR
> specific logic deprecated.
> MRJobStats - Would just extend JobStats.
>
> If we did this, external software could switch their code from using
> JobStats to MRJobStats at their own pace and without breaking against any
> specific version of Pig.  After a few versions the MR specific logic could
> be removed from JobStats and pushed into MRJobStats and it shouldn't break
> anything for people that had made that change.
>
> I'm not familiar with all of the changes in PIG-3419 so this might not work
> everywhere.
>
>
> On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park <[EMAIL PROTECTED]>
> wrote:
>
> > To be specific, we will need to revert all the following commits in
> order:
> >
> >
> > commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982
> > Author: Cheolsoo Park <[EMAIL PROTECTED]>
> > Date:   Fri Sep 20 22:47:29 2013 +0000
> >
> >     PIG-3471: Add a base abstract class for ExecutionEngine (cheolsoo)
> >
> >     git-svn-id:
> >
> >
> https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68
> >
> > commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1
> > Author: Jianyong Dai <[EMAIL PROTECTED]>
> > Date:   Wed Sep 18 19:09:49 2013 +0000
> >
> >     PIG-3457: Provide backward compatibility for PigStatsUtil and
> JobStats
> >
> >     git-svn-id:
> >
> >
> https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68
> >
> > commit e85cf34c92713aa697a1cda7a9c2b3db139350f7
> > Author: Cheolsoo Park <[EMAIL PROTECTED]>
> > Date:   Wed Sep 18 15:37:58 2013 +0000
> >
> >     PIG-3464: Mark ExecType and ExecutionEngine interfaces as evolving
> > (cheolsoo)
> >
> > commit fd8b7cdf9292b305f02386d560c25298ab492a0b
> > Author: Cheolsoo Park <[EMAIL PROTECTED]>
> > Date:   Fri Aug 30 20:04:29 2013 +0000
> >
> >     PIG-3419: Pluggable Execution Engine (achalsoni81 via cheolsoo)
> >
> >     git-svn-id:
> >
> >
> https://svn.apache.org/repos/asf/pig/trunk@151906213f79535-47bb-0310-9956-ffa450edef68
> >
> >
> >
> >
> > On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Thanks Cheolsoo! My opinion is provide backward compatibility for
> > PigStats
> > > is a must, otherwise it could be a havoc. I imagine PigStats is widely
> > used
> > > by Pig users via PigRunner and PPNL interface. People use PigStats to
> > > collect MR job details of the Pig job. Though PigStats is marked for