Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Re: Heads up - 2.0.5-beta


Copy link to this message
-
Re: Heads up - 2.0.5-beta
Luke Lu 2013-04-26, 20:37
If protocol compatibility of v2 and v3 is a goal, HADOOP-8990 should be a
blocker for v2.

__Luke

On Fri, Apr 26, 2013 at 12:07 PM, Eli Collins <[EMAIL PROTECTED]> wrote:

> On Fri, Apr 26, 2013 at 11:15 AM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >
> > On Apr 25, 2013, at 7:31 PM, Roman Shaposhnik wrote:
> >
> >> On Thu, Apr 25, 2013 at 6:34 PM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> With that in mind, I really want to make a serious push to lock down
> APIs and wire-protocols for hadoop-2.0.5-beta.
> >>> Thus, we can confidently support hadoop-2.x in a compatible manner in
> the future. So, it's fine to add new features,
> >>> but please ensure that all APIs are frozen for hadoop-2.0.5-beta
> >>
> >> Arun, since it sounds like you have a pretty definite idea
> >> in mind for what you want 'beta' label to actually mean,
> >> could you, please, share the exact criteria?
> >
> > Sorry, I'm not sure if this is exactly what you are looking for but, as
> I mentioned above, the primary aim would be make the final set of required
> API/write-protocol changes so that we can call it a 'beta' i.e. once
> 2.0.5-beta ships users & downstream projects can be confident about forward
> compatibility in hadoop-2.x line. Obviously, we might discover a blocker
> bug post 2.0.5 which *might* necessitate an unfortunate change - but that
> should be an outstanding exception.
>
> Arun, Suresh,
>
> Mind reviewing the following page Karthik put together on
> compatibility?   http://wiki.apache.org/hadoop/Compatibility
>
> I think we should do something similar to what Sanjay proposed in
> HADOOP-5071 for Hadoop v2.   If we get on the same page on
> compatibility terms/APIs then we can quickly draft the policy, at
> least for the things we've already got consensus on.  I think our new
> developers, users, downstream projects, and partners would really
> appreciate us making this clear.  If people like the content we can
> move it to the Hadoop website and maintain it in svn like the bylaws.
>
> The reason I think we need to do so is because there's been confusion
> about what types of compatibility we promise and some open questions
> which I'm not sure everyone is clear on. Examples:
> - Are we going to preserve Hadoop v3 clients against v2 servers now
> that we have protobuf support?  (I think so..)
> - Can we break rolling upgrade of daemons in updates post GA? (I don't
> think so..)
> - Do we disallow HDFS metadata changes that require an HDFS upgrade in
> an update? (I think so..)
> - Can we remove methods from v2 and v2 updates that were deprecated in
> v0.20-22?  (Unclear)
> - Will we preserve binary compatibility for MR2 going forward? (I think
> so..)
> - Does the ability to support multiple versions of MR simultaneously
> via MR2 change the MR API compatibility story? (I don't think so..)
> - Are the RM protocols sufficiently stable to disallow incompatible
> changes potentially required by non-MR projects? (Unclear, most large
> Yarn deployments I'm aware of are running 0.23, not v2 alphas)
>
> I'm also not sure there's currently consensus on what an incompatible
> change is. For example, I think HADOOP-9151 is incompatible because it
> broke client/server wire compatibility with previous releases and any
> change that breaks wire compatibility is incompatible.  Suresh felt it
> was not an incompatible change because it did not affect API
> compatibility (ie PB is not considered part of the API) and the change
> occurred while v2 is in alpha.  Not sure we need to go through the
> whole exercise of what's allowed in an alpha and beta (water under the
> bridge, hopefully), but I do think we should clearly define an
> incompatible change.  It's fine that v2 has been a bit wild wild west
> in the alpha development stage but I think we need to get a little
> more rigorous.
>
> Thanks,
> Eli
>