I've created a release candidate for hadoop-0.23.0 that I would like to release.

It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.0-rc0/

hadoop-0.23 has significant new features: HDFS Federation and NextGen MapReduce being the highlights.

It's also very early in the 0.23 life-cycle and the release is distinctly 'alpha' quality and is not intended to be stable release right-away. The plan is to deploy this to smallish (500+ nodes) clusters and to start hardening it. This should also help downstream projects (Pig, Hive, HBase) start testing against the 0.23 line. Expect to see more bug-fix releases of the branch-0.23 in the forthcoming weeks (0.23.1 etc.)

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

Arun,

Thanks for rolling a release candidate!

Initial observations:

* Checksums and signature are good.
* The license, notice and readme files are hard to find and are
duplicated. I think they need to be at the top-level.
* There is no changes file for MapReduce.
* The source is not included. This is an absolute requirement for all
Apache releases.
* Are the binary artifacts published to the Maven staging repo? That
would be helpful for testing.
* The tag at http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.0-rc0/
looks good.

Cheers,
Tom

On Tue, Nov 1, 2011 at 2:47 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created a release candidate for hadoop-0.23.0 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.0-rc0/
>
> hadoop-0.23 has significant new features: HDFS Federation and NextGen MapReduce being the highlights.
>
> It's also very early in the 0.23 life-cycle and the release is distinctly 'alpha' quality and is not intended to be stable release right-away. The plan is to deploy this to smallish (500+ nodes) clusters and to start hardening it. This should also help downstream projects (Pig, Hive, HBase) start testing against the 0.23 line. Expect to see more bug-fix releases of the branch-0.23 in the forthcoming weeks (0.23.1 etc.)
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun

I downloaded the bits. Verified the signatures.

Started HDFS daemons in secure mode running as 'hdfs' user, created
directories. Successfully started YARN daemons - ResourceManager and
NodeManager - running as 'mapred' user.

But I am running into an issue while running mapreduce jobs using the
container-executor binary shipped with the tar-ball.

2011-11-01 16:30:00,326 WARN
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
from container is : 2
2011-11-01 16:30:00,326 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
2011-11-01 16:30:00,326 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.io.IOException: App initialization failed (2)
        at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:147)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:859)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
/home/vinodkv/Downloads/hadoop-0.23.0/bin/../bin/container-executor: 1:
Syntax error: Unterminated quoted string

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:261)
        at org.apache.hadoop.util.Shell.run(Shell.java:188)
        at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
        at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:139)
        ... 1 more

Still debugging to see if it is an environment issue.

Thanks,
+Vinod
On Tue, Nov 1, 2011 at 3:17 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> I've created a release candidate for hadoop-0.23.0 that I would like to
> release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.0-rc0/
>
> hadoop-0.23 has significant new features: HDFS Federation and NextGen
> MapReduce being the highlights.
>
> It's also very early in the 0.23 life-cycle and the release is distinctly
> 'alpha' quality and is not intended to be stable release right-away. The
> plan is to deploy this to smallish (500+ nodes) clusters and to start
> hardening it. This should also help downstream projects (Pig, Hive, HBase)
> start testing against the 0.23 line. Expect to see more bug-fix releases of
> the branch-0.23 in the forthcoming weeks (0.23.1 etc.)
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun

On Tue, Nov 1, 2011 at 2:47 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created a release candidate for hadoop-0.23.0 that I would like to release.

Great news!

>This should also help downstream projects (Pig, Hive, HBase) start testing against the 0.23 line.

I would love to help with that type of testing. Question, though: do
you happen to know whether
there are changes available in branches/trunk for Pig, Hive and Mahout
to make these projects
compile against the RC?

Thanks,
Roman.

Nice work Arun, thanks.

A couple comments:

* Regarding version number, any reason we are not using 0.23.0-alpha1
(somebody suggested this last night)?
* Regarding Tom's comment about source not included. My understanding is
that the release should be a source tarball, the -bin tarball is just a
convenience. The -bin tarball should not have to include the source.

Thanks.

Alejandro

On Tue, Nov 1, 2011 at 10:35 AM, Tom White <[EMAIL PROTECTED]> wrote:

> Arun,
>
> Thanks for rolling a release candidate!
>
> Initial observations:
>
> * Checksums and signature are good.
> * The license, notice and readme files are hard to find and are
> duplicated. I think they need to be at the top-level.
> * There is no changes file for MapReduce.
> * The source is not included. This is an absolute requirement for all
> Apache releases.
> * Are the binary artifacts published to the Maven staging repo? That
> would be helpful for testing.
> * The tag at
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.0-rc0/
> looks good.
>
> Cheers,
> Tom
>
> On Tue, Nov 1, 2011 at 2:47 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> > I've created a release candidate for hadoop-0.23.0 that I would like to
> release.
> >
> > It is available at:
> http://people.apache.org/~acmurthy/hadoop-0.23.0-rc0/
> >
> > hadoop-0.23 has significant new features: HDFS Federation and NextGen
> MapReduce being the highlights.
> >
> > It's also very early in the 0.23 life-cycle and the release is
> distinctly 'alpha' quality and is not intended to be stable release
> right-away. The plan is to deploy this to smallish (500+ nodes) clusters
> and to start hardening it. This should also help downstream projects (Pig,
> Hive, HBase) start testing against the 0.23 line. Expect to see more
> bug-fix releases of the branch-0.23 in the forthcoming weeks (0.23.1 etc.)
> >
> > Please try the release and vote; the vote will run for the usual 7 days.
> >
> > thanks,
> > Arun
>

Great work Arun, Thanks a lot!
I will start verifying this from tomorrow.

> > > It's also very early in the 0.23 life-cycle and the release is
> > distinctly 'alpha' quality and is not intended to be stable release
> > right-away.
 Since we know the quality about release, why con't the release name itself alpha as Alejandro or others suggested. This will avoid confusions to the users to pick the stable releases later.

Thanks
Uma

----- Original Message -----
From: Alejandro Abdelnur <[EMAIL PROTECTED]>
Date: Tuesday, November 1, 2011 11:29 pm
Subject: Re: [VOTE] Release hadoop-0.23.0-rc0
To: [EMAIL PROTECTED]

> Nice work Arun, thanks.
>
> A couple comments:
>
> * Regarding version number, any reason we are not using 0.23.0-alpha1
> (somebody suggested this last night)?
> * Regarding Tom's comment about source not included. My
> understanding is
> that the release should be a source tarball, the -bin tarball is
> just a
> convenience. The -bin tarball should not have to include the source.
>
> Thanks.
>
> Alejandro
>
> On Tue, Nov 1, 2011 at 10:35 AM, Tom White <[EMAIL PROTECTED]> wrote:
>
> > Arun,
> >
> > Thanks for rolling a release candidate!
> >
> > Initial observations:
> >
> > * Checksums and signature are good.
> > * The license, notice and readme files are hard to find and are
> > duplicated. I think they need to be at the top-level.
> > * There is no changes file for MapReduce.
> > * The source is not included. This is an absolute requirement for
> all> Apache releases.
> > * Are the binary artifacts published to the Maven staging repo? That
> > would be helpful for testing.
> > * The tag at
> > http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.0-
> rc0/> looks good.
> >
> > Cheers,
> > Tom
> >
> > On Tue, Nov 1, 2011 at 2:47 AM, Arun C Murthy
> <[EMAIL PROTECTED]> wrote:
> > > I've created a release candidate for hadoop-0.23.0 that I would
> like to
> > release.
> > >
> > > It is available at:
> > http://people.apache.org/~acmurthy/hadoop-0.23.0-rc0/
> > >
> > > hadoop-0.23 has significant new features: HDFS Federation and
> NextGen> MapReduce being the highlights.
> > >
> > > It's also very early in the 0.23 life-cycle and the release is
> > distinctly 'alpha' quality and is not intended to be stable release
> > right-away. The plan is to deploy this to smallish (500+ nodes)
> clusters> and to start hardening it. This should also help
> downstream projects (Pig,
> > Hive, HBase) start testing against the 0.23 line. Expect to see more
> > bug-fix releases of the branch-0.23 in the forthcoming weeks
> (0.23.1 etc.)
> > >
> > > Please try the release and vote; the vote will run for the
> usual 7 days.
> > >
> > > thanks,
> > > Arun
> >
>

On Tue, Nov 1, 2011 at 10:57 AM, Alejandro Abdelnur <[EMAIL PROTECTED]>wrote:

> * Regarding version number, any reason we are not using 0.23.0-alpha1
I think it is important to separate the artifact and version number from
the stability assessment (alpha, beta, GA). When we approve a release, it
should generally default to alpha. As confidence grows in the release and
branch, we can vote the stability classification up. The last thing you
want to do is roll a new release to increase a release's stability
classification.

If you haven't read it before, I'd suggest looking at HTTPD's release
guidelines (http://httpd.apache.org/dev/release.html) under "What can I
call this release?"

-- Owen

Owen,

Are we using HTTPD's release guidelines? If read correctly the section you
point out, then our first next stable release would be a 0.24.x. Is it that
what you are suggesting?

Thanks.

Alejandro

On Tue, Nov 1, 2011 at 2:53 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote:

> On Tue, Nov 1, 2011 at 10:57 AM, Alejandro Abdelnur <[EMAIL PROTECTED]
> >wrote:
>
> > * Regarding version number, any reason we are not using 0.23.0-alpha1
>
>
> I think it is important to separate the artifact and version number from
> the stability assessment (alpha, beta, GA). When we approve a release, it
> should generally default to alpha. As confidence grows in the release and
> branch, we can vote the stability classification up. The last thing you
> want to do is roll a new release to increase a release's stability
> classification.
>
> If you haven't read it before, I'd suggest looking at HTTPD's release
> guidelines (http://httpd.apache.org/dev/release.html) under "What can I
> call this release?"
>
> -- Owen
>

-1 This is not a release tarball that folks can vote on, since it
contains no source code.  Please cancel this vote and start a new vote
on an actual source code tarball.

As for the name, if we know for certain that it's alpha quality then
we'd probably be doing users a favor to name it that way.  As the
release stabilizes then we should remove such qualifiers, probably
erring on the side of removing them sooner rather than later, so that we
don't end up with a stable release that still has a qualifier.  But it's
your choice what to call it: if enough folks vote +1 then that's the name.

Doug

On 11/01/2011 02:47 AM, Arun C Murthy wrote:
> I've created a release candidate for hadoop-0.23.0 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.0-rc0/
>
> hadoop-0.23 has significant new features: HDFS Federation and NextGen MapReduce being the highlights.
>
> It's also very early in the 0.23 life-cycle and the release is distinctly 'alpha' quality and is not intended to be stable release right-away. The plan is to deploy this to smallish (500+ nodes) clusters and to start hardening it. This should also help downstream projects (Pig, Hive, HBase) start testing against the 0.23 line. Expect to see more bug-fix releases of the branch-0.23 in the forthcoming weeks (0.23.1 etc.)
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun

On 11/01/2011 03:48 PM, Doug Cutting wrote:
> -1 This is not a release tarball that folks can vote on, since it
> contains no source code.  Please cancel this vote and start a new vote
> on an actual source code tarball.

Okay, I found source code in 8 embedded jars:

% find hadoop-0.23.0 -name '*-sources.jar'
hadoop-0.23.0/hadoop-mapreduce-examples-0.23.0-sources.jar
hadoop-0.23.0/share/hadoop/common/hadoop-common-0.23.0-test-sources.jar
hadoop-0.23.0/share/hadoop/common/hadoop-common-0.23.0-sources.jar
hadoop-0.23.0/share/hadoop/hdfs/hadoop-hdfs-0.23.0-test-sources.jar
hadoop-0.23.0/share/hadoop/hdfs/hadoop-hdfs-0.23.0-sources.jar
hadoop-0.23.0/hadoop-mapreduce-test-0.23.0-sources.jar
hadoop-0.23.0/hadoop-mapreduce-tools-0.23.0-sources.jar
hadoop-0.23.0/hadoop-mapreduce-0.23.0-sources.jar

It's awkward at best to build and run tools like RAT on such a release.
 I'd much prefer a tarball that directly includes a buildable
source-file tree, more-or-less an 'svn export' to vote on.  Folks should
not primarily evaluate binaries when voting.  The ASF primarily produces
and publishes source-code so voting artifacts should be optimized for
evaluation of that.  End-user artifacts should be generated by the
source code.

Doug

On Nov 1, 2011, at 4:36 PM, Doug Cutting wrote:

> On 11/01/2011 03:48 PM, Doug Cutting wrote:
>> -1 This is not a release tarball that folks can vote on, since it
>> contains no source code.  Please cancel this vote and start a new vote
>> on an actual source code tarball.
>
> Okay, I found source code in 8 embedded jars:
>

No worries, I'm cancelling this vote to roll an rc1.

Arun

> % find hadoop-0.23.0 -name '*-sources.jar'
> hadoop-0.23.0/hadoop-mapreduce-examples-0.23.0-sources.jar
> hadoop-0.23.0/share/hadoop/common/hadoop-common-0.23.0-test-sources.jar
> hadoop-0.23.0/share/hadoop/common/hadoop-common-0.23.0-sources.jar
> hadoop-0.23.0/share/hadoop/hdfs/hadoop-hdfs-0.23.0-test-sources.jar
> hadoop-0.23.0/share/hadoop/hdfs/hadoop-hdfs-0.23.0-sources.jar
> hadoop-0.23.0/hadoop-mapreduce-test-0.23.0-sources.jar
> hadoop-0.23.0/hadoop-mapreduce-tools-0.23.0-sources.jar
> hadoop-0.23.0/hadoop-mapreduce-0.23.0-sources.jar
>
> It's awkward at best to build and run tools like RAT on such a release.
> I'd much prefer a tarball that directly includes a buildable
> source-file tree, more-or-less an 'svn export' to vote on.  Folks should
> not primarily evaluate binaries when voting.  The ASF primarily produces
> and publishes source-code so voting artifacts should be optimized for
> evaluation of that.  End-user artifacts should be generated by the
> source code.
>
> Doug

I agree. I'd like to see releases be source first with binary versions provided in parallel where the RM chooses to provide them.

On alphas, I think what is important is that we ship alpha releases, so everyone has something to test and begin integration with. I'd encourage folks to +1 with some flaws, with the expectation that flaws will be fixed later.

On names, Alphas should be clearly labeled on the site.  That is critical. Beyond that I don't think it maters.

---
E14 - typing on glass

On Nov 1, 2011, at 4:36 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> On 11/01/2011 03:48 PM, Doug Cutting wrote:
>> -1 This is not a release tarball that folks can vote on, since it
>> contains no source code.  Please cancel this vote and start a new vote
>> on an actual source code tarball.
>
> Okay, I found source code in 8 embedded jars:
>
> % find hadoop-0.23.0 -name '*-sources.jar'
> hadoop-0.23.0/hadoop-mapreduce-examples-0.23.0-sources.jar
> hadoop-0.23.0/share/hadoop/common/hadoop-common-0.23.0-test-sources.jar
> hadoop-0.23.0/share/hadoop/common/hadoop-common-0.23.0-sources.jar
> hadoop-0.23.0/share/hadoop/hdfs/hadoop-hdfs-0.23.0-test-sources.jar
> hadoop-0.23.0/share/hadoop/hdfs/hadoop-hdfs-0.23.0-sources.jar
> hadoop-0.23.0/hadoop-mapreduce-test-0.23.0-sources.jar
> hadoop-0.23.0/hadoop-mapreduce-tools-0.23.0-sources.jar
> hadoop-0.23.0/hadoop-mapreduce-0.23.0-sources.jar
>
> It's awkward at best to build and run tools like RAT on such a release.
> I'd much prefer a tarball that directly includes a buildable
> source-file tree, more-or-less an 'svn export' to vote on.  Folks should
> not primarily evaluate binaries when voting.  The ASF primarily produces
> and publishes source-code so voting artifacts should be optimized for
> evaluation of that.  End-user artifacts should be generated by the
> source code.
>
> Doug

On Tue, Nov 1, 2011 at 4:55 PM, Eric Baldeschwieler
<[EMAIL PROTECTED]> wrote:
> so everyone has something to test and begin integration with.

Publishing maven artifacts would really help in that regard.

Thanks,
Roman.

Roman,
 We are  publishing the 0.23 snapshot artifacts on
 https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/
(Pig folks are using this) in case you want it ASAP.

 As for the release maven artifacts, Ill let the RM handle it.

thanks
mahadev
On Tue, Nov 1, 2011 at 5:01 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
> On Tue, Nov 1, 2011 at 4:55 PM, Eric Baldeschwieler
> <[EMAIL PROTECTED]> wrote:
>> so everyone has something to test and begin integration with.
>
> Publishing maven artifacts would really help in that regard.
>
> Thanks,
> Roman.
>

On 02/11/11 00:11, Mahadev Konar wrote:
> Roman,
>   We are  publishing the 0.23 snapshot artifacts on
>   https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/
> (Pig folks are using this) in case you want it ASAP.
>
>   As for the release maven artifacts, Ill let the RM handle it.
>
> thanks
> mahadev
>
>

the trouble with the maven snapshots are the artifacts aren't tagged to
a particular date and time; you get rolled forward to whatever is the
latest at a time of maven's choosing. As nobody understands when and why
it makes those decisions, there is no guarantee two people will be using
the same version, which makes filing bugreps all but impossible, which
is why nobody rational should be using them downstream. It's safer to
build and release your own versions.

+1 for putting the RC releases into a public maven/ivy repository

>
>
> On Tue, Nov 1, 2011 at 5:01 PM, Roman Shaposhnik<[EMAIL PROTECTED]>  wrote:
>> On Tue, Nov 1, 2011 at 4:55 PM, Eric Baldeschwieler
>> <[EMAIL PROTECTED]>  wrote:
>>> so everyone has something to test and begin integration with.
>>
>> Publishing maven artifacts would really help in that regard.
>>
>> Thanks,
>> Roman.
>>

Roman,

As of Daniel's checkin this morning (see https://issues.apache.org/jira/browse/PIG-2125) Pig trunk compiles against 0.23.

Alan.

On Nov 1, 2011, at 10:40 AM, Roman Shaposhnik wrote:

> On Tue, Nov 1, 2011 at 2:47 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>> I've created a release candidate for hadoop-0.23.0 that I would like to release.
>
> Great news!
>
>> This should also help downstream projects (Pig, Hive, HBase) start testing against the 0.23 line.
>
> I would love to help with that type of testing. Question, though: do
> you happen to know whether
> there are changes available in branches/trunk for Pig, Hive and Mahout
> to make these projects
> compile against the RC?
>
> Thanks,
> Roman.

On Wed, Nov 2, 2011 at 3:11 PM, Alan Gates <[EMAIL PROTECTED]> wrote:
> Roman,
>
> As of Daniel's checkin this morning (see https://issues.apache.org/jira/browse/PIG-2125) Pig trunk compiles against 0.23.

Perfect! Tiny question: given that you seem to have committed it to
0.10 and 0.9 branches, do
you plan on having a .23 compatible 0.9.2 release any time soon?

Thanks,
Roman.

For some definition of soon. :)

There are still a number of our unit tests that fail with 0.23.  We'd like to get that ironed out.

Alan.

On Nov 2, 2011, at 5:22 PM, Roman Shaposhnik wrote:

> On Wed, Nov 2, 2011 at 3:11 PM, Alan Gates <[EMAIL PROTECTED]> wrote:
>> Roman,
>>
>> As of Daniel's checkin this morning (see https://issues.apache.org/jira/browse/PIG-2125) Pig trunk compiles against 0.23.
>
> Perfect! Tiny question: given that you seem to have committed it to
> 0.10 and 0.9 branches, do
> you plan on having a .23 compatible 0.9.2 release any time soon?
>
> Thanks,
> Roman.