Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop, mail # dev - [DISCUSS] stabilizing Hadoop releases wrt. downstream

Copy link to this message
Re: [DISCUSS] stabilizing Hadoop releases wrt. downstream
Konstantin Boudnik 2013-03-06, 05:02
Great start, Bobby! I certainly can jump on fix something quickly if needed as
well (neither an RE person, but CI is truly a dev. tool!)


On Tue, Mar 05, 2013 at 07:18AM, Robert Evans wrote:
> That is a great point.  I have been meaning to set up the Jenkins build
> for branch-2 for a while, so I took the 10 mins and just did it.
> https://builds.apache.org/job/Hadoop-Common-2-Commit/
> Don't let the name fool you, it publishes not just common, but HDFS, YARN,
> MR, and tools too.  You should now have branch-2 SNAPSHOTS updated on each
> commit to branch-2.  Feel free to bug me if you need more integration
> points.  I am not an RE guy, but I can hack it to make things work :)
> --Bobby
> On 3/5/13 12:15 AM, "Konstantin Boudnik" <[EMAIL PROTECTED]> wrote:
> >Arun,
> >
> >first of all, I don't think anyone is trying to put a blame on someone
> >else. E.g. I had similar experience with Oozie being broken because of
> >certain released changes in the upstream.
> >
> >I am sure that most people in BigTop community - especially those who
> >share the committer-ship privilege in BigTop and other upstream
> >projects, including Hadoop, - would be happy to help with the
> >stabilization of the Hadoop base. The issue that a downstream
> >integration project is likely to have is - for once - the absence of
> >regularly published development artifacts. In the light of "it didn't
> >happen if there's no picture" here's a couple of examples:
> >
> >  - 2.0.2-SNAPSHOT weren't published at all; only release 2.0.2-alpha
> >artifacts were
> >  - 2.0.3-SNAPSHOT weren't published until Feb 29, 2013 (it happened just
> >once)
> >
> >So, technically speaking, unless an integration project is willing to
> >build and maintain its own artifacts, it is impossible to do any
> >preventive validation.
> >
> >Which brings me to my next question: how do you guys address
> >"Integration is high on the list of *every* release". Again, please
> >don't get me wrong - I am not looking to lay a blame on or corner
> >anyone - I am really curious and would appreciate the input.
> >
> >
> >Vinod:
> >
> >> As you yourself noted later, the pain is part of the 'alpha' status
> >> of the release. We are targeting +one of the immediate future
> >> releases to be a beta and so these troubles are really only the
> >> short +term.
> >
> >I don't really want to get into the discussion about of what
> >constitutes the alpha and how it has delayed the adoption of Hadoop2
> >line. However, I want to point out that it is especially important for
> >"alpha" platform to work nicely with downstream consumers of the said
> >platform. For quite obvious reasons, I believe.
> >
> >> I think there is a fundamental problem with the interaction of
> >> Bigtop with the downstream projects, if nothing else, with
> >
> >BigTop is as downstream as it can get, because BigTop essentially
> >consumes all other component releases in order to produce a viable
> >stack. Technicalities aside...
> >
> >> Hadoop. We never formalized on the process, will BigTop step in
> >> after an RC is up for vote or before? As I see it, it's happening
> >
> >Bigtop essentially can give any component, including Hadoop, and
> >better yet - the set of components - certain guaratees about
> >compatibility and dependencies being included. Case in point is
> >missing commons libraries missed in 1.0.1 release that essentially
> >prevented HBase from working properly.
> >
> >> after the vote is up, so no wonder we are in this state. Shall we
> >> have a pre-notice to Bigtop so that it can step in before?
> >
> >The above is in contradiction with earlier statement of "Integration
> >is high on the list of *every* release". If BigTop isn't used for
> >integration testing, then how said integration testing is performed?
> >Is it some sort of test-patch process as Luke referred earlier?  And
> >why it leaves the room for the integration issues being uncaught?
> >Again, I am genuinely interested to know.