Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Bigtop, mail # dev - [DISCUSS] stabilizing Hadoop releases wrt. downstream


+
Roman Shaposhnik 2013-02-27, 01:31
+
Roman Shaposhnik 2013-02-27, 01:43
+
Arun C Murthy 2013-03-01, 18:58
+
Konstantin Boudnik 2013-03-05, 06:15
+
Robert Evans 2013-03-05, 15:18
+
Konstantin Boudnik 2013-03-06, 05:02
Copy link to this message
-
Re: [DISCUSS] stabilizing Hadoop releases wrt. downstream
Giridharan Kesavan 2013-03-06, 06:05
Thanks Bobby. I 've setup a unit test
job<https://builds.apache.org/view/Hadoop/job/Hadoop-Common-2-Build/>to
execute unit test's on branch-2 on a daily basis.
I'm happy to help with build setup. Let me know.

-Giri
On Tue, Mar 5, 2013 at 9:02 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> Great start, Bobby! I certainly can jump on fix something quickly if
> needed as
> well (neither an RE person, but CI is truly a dev. tool!)
>
> Thanks!
>   Cos
>
> On Tue, Mar 05, 2013 at 07:18AM, Robert Evans wrote:
> > That is a great point.  I have been meaning to set up the Jenkins build
> > for branch-2 for a while, so I took the 10 mins and just did it.
> >
> > https://builds.apache.org/job/Hadoop-Common-2-Commit/
> >
> > Don't let the name fool you, it publishes not just common, but HDFS,
> YARN,
> > MR, and tools too.  You should now have branch-2 SNAPSHOTS updated on
> each
> > commit to branch-2.  Feel free to bug me if you need more integration
> > points.  I am not an RE guy, but I can hack it to make things work :)
> >
> > --Bobby
> >
> > On 3/5/13 12:15 AM, "Konstantin Boudnik" <[EMAIL PROTECTED]> wrote:
> >
> > >Arun,
> > >
> > >first of all, I don't think anyone is trying to put a blame on someone
> > >else. E.g. I had similar experience with Oozie being broken because of
> > >certain released changes in the upstream.
> > >
> > >I am sure that most people in BigTop community - especially those who
> > >share the committer-ship privilege in BigTop and other upstream
> > >projects, including Hadoop, - would be happy to help with the
> > >stabilization of the Hadoop base. The issue that a downstream
> > >integration project is likely to have is - for once - the absence of
> > >regularly published development artifacts. In the light of "it didn't
> > >happen if there's no picture" here's a couple of examples:
> > >
> > >  - 2.0.2-SNAPSHOT weren't published at all; only release 2.0.2-alpha
> > >artifacts were
> > >  - 2.0.3-SNAPSHOT weren't published until Feb 29, 2013 (it happened
> just
> > >once)
> > >
> > >So, technically speaking, unless an integration project is willing to
> > >build and maintain its own artifacts, it is impossible to do any
> > >preventive validation.
> > >
> > >Which brings me to my next question: how do you guys address
> > >"Integration is high on the list of *every* release". Again, please
> > >don't get me wrong - I am not looking to lay a blame on or corner
> > >anyone - I am really curious and would appreciate the input.
> > >
> > >
> > >Vinod:
> > >
> > >> As you yourself noted later, the pain is part of the 'alpha' status
> > >> of the release. We are targeting +one of the immediate future
> > >> releases to be a beta and so these troubles are really only the
> > >> short +term.
> > >
> > >I don't really want to get into the discussion about of what
> > >constitutes the alpha and how it has delayed the adoption of Hadoop2
> > >line. However, I want to point out that it is especially important for
> > >"alpha" platform to work nicely with downstream consumers of the said
> > >platform. For quite obvious reasons, I believe.
> > >
> > >> I think there is a fundamental problem with the interaction of
> > >> Bigtop with the downstream projects, if nothing else, with
> > >
> > >BigTop is as downstream as it can get, because BigTop essentially
> > >consumes all other component releases in order to produce a viable
> > >stack. Technicalities aside...
> > >
> > >> Hadoop. We never formalized on the process, will BigTop step in
> > >> after an RC is up for vote or before? As I see it, it's happening
> > >
> > >Bigtop essentially can give any component, including Hadoop, and
> > >better yet - the set of components - certain guaratees about
> > >compatibility and dependencies being included. Case in point is
> > >missing commons libraries missed in 1.0.1 release that essentially
> > >prevented HBase from working properly.
> > >
> > >> after the vote is up, so no wonder we are in this state. Shall we
> > >> have a pre-notice to Bigtop so that it can step in before?
+
Arun C Murthy 2013-03-06, 15:24
+
Konstantin Boudnik 2013-03-06, 18:19
+
Roman Shaposhnik 2013-03-08, 17:55
+
Matt Foley 2013-03-08, 22:16
+
Vinod Kumar Vavilapalli 2013-03-01, 20:23