Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - [Discussion] Any thoughts on PIG-3457?


Copy link to this message
-
Re: [Discussion] Any thoughts on PIG-3457?
Rohini Palaniswamy 2013-09-30, 21:26
+1. I was already asking for keeping the new API changes only in Tez branch
till it evolves and is finalized, so I have no objections to reverting it.

Regards,
Rohini
On Mon, Sep 30, 2013 at 1:28 PM, Alan Gates <[EMAIL PROTECTED]> wrote:

> We should separate out two separate concerns.  If I understand correctly
> we don't need any of these changes in 0.12.  So we should revert these
> patches from the 12 branch so that we can get it released quickly in a
> backwards compatible way.
>
> We will then have plenty of time to discuss the separate question of how
> we proceed going forward (deprecated APIs or new APIs).
>
> Alan.
>
> On Sep 30, 2013, at 11:45 AM, Cheolsoo Park wrote:
>
> > Hi Jeremy,
> >
> > What you're saying makes sense, and patch is welcome. ;-) But complexity
> > comes from that there are many classes that are associated with one
> > another, and it seems necessary to bring back all of them together in
> order
> > to provide full backward compatibility.
> >
> > After spending many hours on the weekend, I concluded that adding more
> > workarounds (classes, methods, packages, etc) to the current code makes
> it
> > only less maintainable and readable. So I prefer a simpler approach.
> >
> > For eg, we can just publish two jars - pig.jar w/ old API and pig-new.jar
> > w/ new API - maybe not in 0.12 but in 0.13. Since we already have a
> > tez-branch, we can use it to manage the new version of classes. Then,
> users
> > can switch to pig-new.jar gradually in 0.13 and 0.14. When we finally
> merge
> > tez-branch into trunk, we can publish a single jar again.
> >
> > Of course, this is not trivial either because we have to maintain two
> > branches. But I feel that managing two branches independently is easier
> > than maintaining all sorts of workarounds for backward compatibility in
> the
> > source code. In addition, we will have more flexibility in terms of
> > designing new API because we will be completely free from backward
> > compatibility. No?
> >
> > Thanks,
> > Cheolsoo
> >
> > On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn <[EMAIL PROTECTED]>
> wrote:
> >
> >> What about the option of leaving all of the MR specific logic in the
> >> original classes but marking those methods as deprecated and telling
> people
> >> to switch to using a MR specific object that extends the original class.
> >> So for example:
> >>
> >> JobStats - Reverted to being as it was before PIG-3419 but with all MR
> >> specific logic deprecated.
> >> MRJobStats - Would just extend JobStats.
> >>
> >> If we did this, external software could switch their code from using
> >> JobStats to MRJobStats at their own pace and without breaking against
> any
> >> specific version of Pig.  After a few versions the MR specific logic
> could
> >> be removed from JobStats and pushed into MRJobStats and it shouldn't
> break
> >> anything for people that had made that change.
> >>
> >> I'm not familiar with all of the changes in PIG-3419 so this might not
> work
> >> everywhere.
> >>
> >>
> >> On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >>> To be specific, we will need to revert all the following commits in
> >> order:
> >>>
> >>>
> >>> commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982
> >>> Author: Cheolsoo Park <[EMAIL PROTECTED]>
> >>> Date:   Fri Sep 20 22:47:29 2013 +0000
> >>>
> >>>    PIG-3471: Add a base abstract class for ExecutionEngine (cheolsoo)
> >>>
> >>>    git-svn-id:
> >>>
> >>>
> >>
> https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68
> >>>
> >>> commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1
> >>> Author: Jianyong Dai <[EMAIL PROTECTED]>
> >>> Date:   Wed Sep 18 19:09:49 2013 +0000
> >>>
> >>>    PIG-3457: Provide backward compatibility for PigStatsUtil and
> >> JobStats
> >>>
> >>>    git-svn-id:
> >>>
> >>>
> >>
> https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68
> >>>
> >>> commit e85cf34c92713aa697a1cda7a9c2b3db139350f7