Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> [Discussion] Any thoughts on PIG-3457?

Copy link to this message
Re: [Discussion] Any thoughts on PIG-3457?
To be specific, we will need to revert all the following commits in order:
commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982
Author: Cheolsoo Park <[EMAIL PROTECTED]>
Date:   Fri Sep 20 22:47:29 2013 +0000

    PIG-3471: Add a base abstract class for ExecutionEngine (cheolsoo)


commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1
Author: Jianyong Dai <[EMAIL PROTECTED]>
Date:   Wed Sep 18 19:09:49 2013 +0000

    PIG-3457: Provide backward compatibility for PigStatsUtil and JobStats


commit e85cf34c92713aa697a1cda7a9c2b3db139350f7
Author: Cheolsoo Park <[EMAIL PROTECTED]>
Date:   Wed Sep 18 15:37:58 2013 +0000

    PIG-3464: Mark ExecType and ExecutionEngine interfaces as evolving

commit fd8b7cdf9292b305f02386d560c25298ab492a0b
Author: Cheolsoo Park <[EMAIL PROTECTED]>
Date:   Fri Aug 30 20:04:29 2013 +0000

    PIG-3419: Pluggable Execution Engine (achalsoni81 via cheolsoo)

On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai <[EMAIL PROTECTED]> wrote:

> Thanks Cheolsoo! My opinion is provide backward compatibility for PigStats
> is a must, otherwise it could be a havoc. I imagine PigStats is widely used
> by Pig users via PigRunner and PPNL interface. People use PigStats to
> collect MR job details of the Pig job. Though PigStats is marked for
> Evolving, this is mostly for extending PigStats, not limiting PigStats as
> PIG-3419 did. Even if we really need to change, we need to very well
> communicate with users over time, Pig 0.12 is not an option.
> PIG-3457 is trying to provide a backward compatibility way for PigStats,
> but just like Cheolsoo said, it is far from ideal. I now tend to agree
> Rohini's suggestion on PIG-3419, rollback PIG-3419, until we find a better
> way. Seems PIG-3419 is a little premature. Besides the above mentioned
> PigStats issue, I've already found 2 additional issues:
> 1. "explain" shows unoptimized logical plan instead of optimized one
> 2. HangingJobKiller is removed
> How does others think?
> Thanks,
> Daniel
> On Mon, Sep 30, 2013 at 9:51 AM, Cheolsoo Park <[EMAIL PROTECTED]>
> wrote:
> > Hi devs,
> >
> > PIG-3419 <https://issues.apache.org/jira/browse/PIG-3419> broke backward
> > compatibility for downstream applications such as Oozie, and
> > PIG-3457<https://issues.apache.org/jira/browse/PIG-3457> is
> > trying to fix it.
> >
> > In summary, we need to keep the old MR-specific JobStats and PigStats for
> > backward compatibility sadly because they're currently exposed in several
> > user-facing API including PigRunner, PigServer, etc. However, this
> defeats
> > the purpose of PIG-3419
> > <https://issues.apache.org/jira/browse/PIG-3419> because
> > we cannot implement non-MR execution engines in the back-end if the
> > front-end API is tied to MR.
> >
> > Daniel and I were having a discussion in the jira, but it seems more
> > complicated than I thought. I am wondering whether anyone has a good
> > suggestion on how to solve it. This is blocking the 0.12 release.
> >
> > Thanks,
> > Cheolsoo
> >
> --
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.