Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - [Discussion] Any thoughts on PIG-3457?


Copy link to this message
-
Re: [Discussion] Any thoughts on PIG-3457?
Jeremy Karn 2013-09-30, 19:56
Ok, sounds good.  I'll take a shot at it tonight.
On Mon, Sep 30, 2013 at 3:48 PM, Daniel Dai <[EMAIL PROTECTED]> wrote:

> Thanks Jeremy. That sounds absolutely fine. The only reservation is I don't
> want to delay 0.12.0 release. We need to either do it quickly, or rollback
> PIG-3419 and then do it on 0.13.
>
> Thanks,
> Daniel
>
>
> On Mon, Sep 30, 2013 at 12:40 PM, Jeremy Karn <[EMAIL PROTECTED]>
> wrote:
>
> > I don't mind trying to put together a patch for what I described above if
> > there's a general consensus on the strategy we should take (or at least
> no
> > big objections).
> >
> > I think the multiple jars solution could be troublesome, but maybe after
> > seeing what a patch looks like for a single jar solution it'll seem
> nicer.
> >
> >
> > On Mon, Sep 30, 2013 at 2:45 PM, Cheolsoo Park <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi Jeremy,
> > >
> > > What you're saying makes sense, and patch is welcome. ;-) But
> complexity
> > > comes from that there are many classes that are associated with one
> > > another, and it seems necessary to bring back all of them together in
> > order
> > > to provide full backward compatibility.
> > >
> > > After spending many hours on the weekend, I concluded that adding more
> > > workarounds (classes, methods, packages, etc) to the current code makes
> > it
> > > only less maintainable and readable. So I prefer a simpler approach.
> > >
> > > For eg, we can just publish two jars - pig.jar w/ old API and
> pig-new.jar
> > > w/ new API - maybe not in 0.12 but in 0.13. Since we already have a
> > > tez-branch, we can use it to manage the new version of classes. Then,
> > users
> > > can switch to pig-new.jar gradually in 0.13 and 0.14. When we finally
> > merge
> > > tez-branch into trunk, we can publish a single jar again.
> > >
> > > Of course, this is not trivial either because we have to maintain two
> > > branches. But I feel that managing two branches independently is easier
> > > than maintaining all sorts of workarounds for backward compatibility in
> > the
> > > source code. In addition, we will have more flexibility in terms of
> > > designing new API because we will be completely free from backward
> > > compatibility. No?
> > >
> > > Thanks,
> > > Cheolsoo
> > >
> > > On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > What about the option of leaving all of the MR specific logic in the
> > > > original classes but marking those methods as deprecated and telling
> > > people
> > > > to switch to using a MR specific object that extends the original
> > class.
> > > >  So for example:
> > > >
> > > > JobStats - Reverted to being as it was before PIG-3419 but with all
> MR
> > > > specific logic deprecated.
> > > > MRJobStats - Would just extend JobStats.
> > > >
> > > > If we did this, external software could switch their code from using
> > > > JobStats to MRJobStats at their own pace and without breaking against
> > any
> > > > specific version of Pig.  After a few versions the MR specific logic
> > > could
> > > > be removed from JobStats and pushed into MRJobStats and it shouldn't
> > > break
> > > > anything for people that had made that change.
> > > >
> > > > I'm not familiar with all of the changes in PIG-3419 so this might
> not
> > > work
> > > > everywhere.
> > > >
> > > >
> > > > On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > To be specific, we will need to revert all the following commits in
> > > > order:
> > > > >
> > > > >
> > > > > commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982
> > > > > Author: Cheolsoo Park <[EMAIL PROTECTED]>
> > > > > Date:   Fri Sep 20 22:47:29 2013 +0000
> > > > >
> > > > >     PIG-3471: Add a base abstract class for ExecutionEngine
> > (cheolsoo)
> > > > >
> > > > >     git-svn-id:
> > > > >
> > > > >
> > > >
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68
> > >

Jeremy Karn / Lead Developer
MORTAR DATA / 519 277 4391 / www.mortardata.com