Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # general >> Streamlining the Hadoop release process


+
Konstantin Shvachko 2013-04-25, 03:17
+
Roman Shaposhnik 2013-04-25, 15:44
+
Konstantin Shvachko 2013-04-25, 22:41
+
Robert Evans 2013-04-25, 15:28
+
Roman Shaposhnik 2013-04-25, 15:48
+
Robert Evans 2013-04-25, 22:05
+
Konstantin Shvachko 2013-04-25, 22:06
+
Robert Evans 2013-04-29, 18:53
+
Roman Shaposhnik 2013-04-30, 04:34
Copy link to this message
-
Re: Streamlining the Hadoop release process
We are running nightly tests on branch-2 and branch-0.23.  These include
unit and integration tests and branch-2 seems fairly good.  We would like
to move off of 0.23 to 2.0.  But, there are relatively few things in 2.0
that really push us in that direction over what we already have in 0.23.
The major ones are HDFS HA and RM recovery as steps towards rolling
upgrades.  For 2.0.5 to be beta for me really requires that the APIs are
locked down, for both wire and binary compatibility.  This fits with what
Arun has stated as his intentions and also with the compatibility
discussion that has been going on. All of this is great but also requires
a strong commitment from the entire community to really make this happen.
I wish I could have participated more in those discussions but with
everything else that has been on my plate I have not even had enough time
to read them in a timely fashion.  For us to adopt it widely we need 2.0.X
to be stable at scale.  We have worked out most of the issues on YARN,
there may be a few more because of the deltas between 0.23 and 2.0 but
that should be relatively small.  The big question is with HDFS.  The base
HA work seems to be fairly stable with the fuzz testing and everything
else that has happened, but I just don't know how it is going to handle a
4000 node cluster. I don't know if BigTop can help with that, but any type
of stress testing is desirable.  The other HDFS features like snapshots
just seem to add more risk for something that we do not see using in the
short term.  Not that it is a bad feature, we just don't need it yet.

--Bobby

On 4/29/13 11:34 PM, "Roman Shaposhnik" <[EMAIL PROTECTED]> wrote:

>On Mon, Apr 29, 2013 at 11:53 AM, Robert Evans <[EMAIL PROTECTED]>
>wrote:
>> This depends on how quickly Yahoo will be off of branch-0.23 and if
>>Yahoo
>> is the only one maintaining it then it will not receive any bug fixes.
>> Does that make branch-0.23 any less stable, no.  Does it make it less
>> desirable for people to move to? Yes.
>
>Do you think there's anything that we can do as a community to help with
>that?
>Or on a flip side -- do you think there's any chance for Yahoo! team
>to formalize
>the kind of workflows that the company cares about in Bigtop integration
>or
>component unit tests?
>
>Personally, I'm mostly nervous about the fact that I'm yet to hear any of
>the
>DevOps from major Hadoop using companies to chime in on the 2.0.5-beta
>discussion with their take on what 'beta' means to them.
>
>Thanks,
>Roman.
+
Konstantin Boudnik 2013-05-01, 05:40
+
Shv.hadoop 2013-05-01, 06:51