I have created a candidate build for Hadoop 0.21.0. This fixes over
one thousand issues in Common, HDFS, and MapReduce since 0.20.2.

This release is being classified as a minor release, which means that
it is API compatible with 0.20.2.

*** This release candidate has not been tested extensively, so it
should not be considered stable.
*** Please download, test and vote before the vote closes on Thursday 8 July.

http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/

The hadoop-0.21.0.tar.gz file is an old-style combined release which
includes Common, HDFS, and MapReduce.

Thanks,
Tom

P.S. I have not yet pushed the JARs to Apache's Maven Staging
Repository (see HADOOP-6847), but this shouldn't prevent testing and
voting while it is sorted out.

A problem was found with Job.getCounters() returning null (see
https://issues.apache.org/jira/browse/MAPREDUCE-1920). I plan to roll
a new release candidate when this is fixed.

Cheers,
Tom

On Fri, Jul 2, 2010 at 8:25 AM, Tom White <[EMAIL PROTECTED]> wrote:
> I have created a candidate build for Hadoop 0.21.0. This fixes over
> one thousand issues in Common, HDFS, and MapReduce since 0.20.2.
>
> This release is being classified as a minor release, which means that
> it is API compatible with 0.20.2.
>
> *** This release candidate has not been tested extensively, so it
> should not be considered stable.
> *** Please download, test and vote before the vote closes on Thursday 8 July.
>
> http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/
>
> The hadoop-0.21.0.tar.gz file is an old-style combined release which
> includes Common, HDFS, and MapReduce.
>
> Thanks,
> Tom
>
> P.S. I have not yet pushed the JARs to Apache's Maven Staging
> Repository (see HADOOP-6847), but this shouldn't prevent testing and
> voting while it is sorted out.
>

Hi Tom,

I'm trying to build Hadoop 0.21.0 locally, but its failing because the
hadoop root dir is missing the build.xml in the tar.gz. Is there a new
build process ? I'm currently using the ant clean tar test-core
directives.

Regards
Steve Watt

From:
Tom White <[EMAIL PROTECTED]>
To:
common-dev <[EMAIL PROTECTED]>
Date:
07/08/2010 07:16 AM
Subject:
Re: [VOTE] Release Hadoop 0.21.0 (candidate 0)

A problem was found with Job.getCounters() returning null (see
https://issues.apache.org/jira/browse/MAPREDUCE-1920). I plan to roll
a new release candidate when this is fixed.

Cheers,
Tom

On Fri, Jul 2, 2010 at 8:25 AM, Tom White <[EMAIL PROTECTED]> wrote:
> I have created a candidate build for Hadoop 0.21.0. This fixes over
> one thousand issues in Common, HDFS, and MapReduce since 0.20.2.
>
> This release is being classified as a minor release, which means that
> it is API compatible with 0.20.2.
>
> *** This release candidate has not been tested extensively, so it
> should not be considered stable.
> *** Please download, test and vote before the vote closes on Thursday 8
July.
>
> http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/
>
> The hadoop-0.21.0.tar.gz file is an old-style combined release which
> includes Common, HDFS, and MapReduce.
>
> Thanks,
> Tom
>
> P.S. I have not yet pushed the JARs to Apache's Maven Staging
> Repository (see HADOOP-6847), but this shouldn't prevent testing and
> voting while it is sorted out.
>

I downloaded the "combined" tarball of 0.21.0-rc0 and set it up as a
pseudo-distributed Hadoop cluster.

Everything seems to work; basic smoke tests pass. Did not run internal unit
tests. I tested Sqoop 1.0.0 against this release. All sqoop unit tests pass.
Sqoop can operate on the command-line as a "normal" Hadoop application
against 0.21.0 successfully as long as MAPREDUCE-1920 is applied (or the
appropriate configuration changes are made in mapred-site.xml).

The other problem I found is that build-utils.xml is missing which prevents
recompilation.

I am a (nonbinding) +1 to this release candidate along with modifications to
include MR-1920 and fix the missing build-utils.xml issue.

- Aaron

On Fri, Jul 2, 2010 at 12:25 AM, Tom White <[EMAIL PROTECTED]> wrote:

> I have created a candidate build for Hadoop 0.21.0. This fixes over
> one thousand issues in Common, HDFS, and MapReduce since 0.20.2.
>
> This release is being classified as a minor release, which means that
> it is API compatible with 0.20.2.
>
> *** This release candidate has not been tested extensively, so it
> should not be considered stable.
> *** Please download, test and vote before the vote closes on Thursday 8
> July.
>
> http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/
>
> The hadoop-0.21.0.tar.gz file is an old-style combined release which
> includes Common, HDFS, and MapReduce.
>
> Thanks,
> Tom
>
> P.S. I have not yet pushed the JARs to Apache's Maven Staging
> Repository (see HADOOP-6847), but this shouldn't prevent testing and
> voting while it is sorted out.
>

A number of issues have been identified with RC0. The following JIRAs
need fixing before the next release candidate can be built:

HDFS-1288 - start-all.sh / stop-all.sh does not seem to work with HDFS
MAPREDUCE-1920 - Job.getCounters() returns null when using a cluster
MAPREDUCE-1926 - MapReduce distribution is missing build-utils.xml

In addition, the following JIRAs need fixing before the release is finalized:

HDFS-1292 - Allow artifacts to be published to the staging Apache
Nexus Maven Repository
MAPREDUCE-1929 - Allow artifacts to be published to the staging Apache
Nexus Maven Repository

Tom

On Thu, Jul 8, 2010 at 7:55 PM, Aaron Kimball <[EMAIL PROTECTED]> wrote:
> I downloaded the "combined" tarball of 0.21.0-rc0 and set it up as a
> pseudo-distributed Hadoop cluster.
>
> Everything seems to work; basic smoke tests pass. Did not run internal unit
> tests. I tested Sqoop 1.0.0 against this release. All sqoop unit tests pass.
> Sqoop can operate on the command-line as a "normal" Hadoop application
> against 0.21.0 successfully as long as MAPREDUCE-1920 is applied (or the
> appropriate configuration changes are made in mapred-site.xml).
>
> The other problem I found is that build-utils.xml is missing which prevents
> recompilation.
>
> I am a (nonbinding) +1 to this release candidate along with modifications to
> include MR-1920 and fix the missing build-utils.xml issue.
>
> - Aaron
>
> On Fri, Jul 2, 2010 at 12:25 AM, Tom White <[EMAIL PROTECTED]> wrote:
>
>> I have created a candidate build for Hadoop 0.21.0. This fixes over
>> one thousand issues in Common, HDFS, and MapReduce since 0.20.2.
>>
>> This release is being classified as a minor release, which means that
>> it is API compatible with 0.20.2.
>>
>> *** This release candidate has not been tested extensively, so it
>> should not be considered stable.
>> *** Please download, test and vote before the vote closes on Thursday 8
>> July.
>>
>> http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/
>>
>> The hadoop-0.21.0.tar.gz file is an old-style combined release which
>> includes Common, HDFS, and MapReduce.
>>
>> Thanks,
>> Tom
>>
>> P.S. I have not yet pushed the JARs to Apache's Maven Staging
>> Repository (see HADOOP-6847), but this shouldn't prevent testing and
>> voting while it is sorted out.
>>
>

This is likely a result of how things are now being built post
project-split, but previously, for the hadoop-0.20.x releases there was a
top level build.xml file which would orchestrate building the sub-projects
which were split underneath the src directory, resulting in a final
hadoop-20.x-core.jar. I see now each sub-project (common, hdfs and mapred)
has its own subdirectory off of the root, as opposed to forking under the
src/ directory and in turn their own sub-project jar files and there is
now no aggregated hadoop-x-core.jar file.

I also saw Tom's JIRA with the 0.21 build scripts. My questions are:

1) Is this the way we're going to be building going forward for the
foreseeable future?
2) Could we consider still keeping a top level build file (i.e committing
a variant of tom's scripts) that orchestrates the building of all the
sub-projects?

Kind regards
Steve Watt

Hi Tom,

Just want to let you know that back then when I tried to circumvent
this problem, I used job.submit() and retrieve the counter
asynchronously. I found out that the counter values always zero during
the execution (so the job is still running halfway). So, in this case,
the job is not retired yet, right? so, it might suggest another
problem?

Felix Halim

On Thu, Jul 8, 2010 at 8:13 PM, Tom White <[EMAIL PROTECTED]> wrote:
> A problem was found with Job.getCounters() returning null (see
> https://issues.apache.org/jira/browse/MAPREDUCE-1920). I plan to roll
> a new release candidate when this is fixed.
>
> Cheers,
> Tom
>
> On Fri, Jul 2, 2010 at 8:25 AM, Tom White <[EMAIL PROTECTED]> wrote:
>> I have created a candidate build for Hadoop 0.21.0. This fixes over
>> one thousand issues in Common, HDFS, and MapReduce since 0.20.2.
>>
>> This release is being classified as a minor release, which means that
>> it is API compatible with 0.20.2.
>>
>> *** This release candidate has not been tested extensively, so it
>> should not be considered stable.
>> *** Please download, test and vote before the vote closes on Thursday 8 July.
>>
>> http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/
>>
>> The hadoop-0.21.0.tar.gz file is an old-style combined release which
>> includes Common, HDFS, and MapReduce.
>>
>> Thanks,
>> Tom
>>
>> P.S. I have not yet pushed the JARs to Apache's Maven Staging
>> Repository (see HADOOP-6847), but this shouldn't prevent testing and
>> voting while it is sorted out.
>>
>

Steve, Yes, this is a consequence of the project split. It's still
possible to build Common, HDFS, MapReduce individually (from their
respective distributions), but I agree it would be nice to be able to
build from the combined distribution. When I create the next RC, I'll
see if I can provide instructions on how to do this.

Felix, I don't think this is a bug since counters are not stable until
the job is complete, so the values you see may be zero if you retrieve
them asynchronously.

Cheers,
Tom

PS. I'm travelling/on vacation for the next couple of weeks, so I
won't be working on the next release candidate until August. If anyone
wants to work on the (blocker) JIRAs I listed in my previous email in
this thread then be my guest!

On Tue, Jul 13, 2010 at 12:04 AM, Felix Halim <[EMAIL PROTECTED]> wrote:
> Hi Tom,
>
> Just want to let you know that back then when I tried to circumvent
> this problem, I used job.submit() and retrieve the counter
> asynchronously. I found out that the counter values always zero during
> the execution (so the job is still running halfway). So, in this case,
> the job is not retired yet, right? so, it might suggest another
> problem?
>
> Felix Halim
>
> On Thu, Jul 8, 2010 at 8:13 PM, Tom White <[EMAIL PROTECTED]> wrote:
>> A problem was found with Job.getCounters() returning null (see
>> https://issues.apache.org/jira/browse/MAPREDUCE-1920). I plan to roll
>> a new release candidate when this is fixed.
>>
>> Cheers,
>> Tom
>>
>> On Fri, Jul 2, 2010 at 8:25 AM, Tom White <[EMAIL PROTECTED]> wrote:
>>> I have created a candidate build for Hadoop 0.21.0. This fixes over
>>> one thousand issues in Common, HDFS, and MapReduce since 0.20.2.
>>>
>>> This release is being classified as a minor release, which means that
>>> it is API compatible with 0.20.2.
>>>
>>> *** This release candidate has not been tested extensively, so it
>>> should not be considered stable.
>>> *** Please download, test and vote before the vote closes on Thursday 8 July.
>>>
>>> http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/
>>>
>>> The hadoop-0.21.0.tar.gz file is an old-style combined release which
>>> includes Common, HDFS, and MapReduce.
>>>
>>> Thanks,
>>> Tom
>>>
>>> P.S. I have not yet pushed the JARs to Apache's Maven Staging
>>> Repository (see HADOOP-6847), but this shouldn't prevent testing and
>>> voting while it is sorted out.
>>>
>>
>

On Fri, Jul 16, 2010 at 6:17 PM, Tom White <[EMAIL PROTECTED]> wrote:
> Felix, I don't think this is a bug since counters are not stable until
> the job is complete, so the values you see may be zero if you retrieve
> them asynchronously.

Just to clarify.. these counters that are retrieved asynchronously
will be updated as soon as one mapper or reducer completes, right?
Or is it going to be delayed arbitrarily until the job is finished?

BTW, I have a warning when running a job:

10/07/20 17:44:19 WARN conf.Configuration: mapred.task.id is
deprecated. Instead, use mapreduce.task.attempt.id

What causes this warning? I don't use mapred.task.id anywhere in the
configurations...

Felix Halim

On Tue, Jul 20, 2010 at 10:50 AM, Felix Halim <[EMAIL PROTECTED]> wrote:
> On Fri, Jul 16, 2010 at 6:17 PM, Tom White <[EMAIL PROTECTED]> wrote:
>> Felix, I don't think this is a bug since counters are not stable until
>> the job is complete, so the values you see may be zero if you retrieve
>> them asynchronously.
>
> Just to clarify.. these counters that are retrieved asynchronously
> will be updated as soon as one mapper or reducer completes, right?
> Or is it going to be delayed arbitrarily until the job is finished?

To be clear, the counters objects will not be updated, you have to
retrieve them again, and you can only be sure they are correct after
the job completes.

>
> BTW, I have a warning when running a job:
>
> 10/07/20 17:44:19 WARN conf.Configuration: mapred.task.id is
> deprecated. Instead, use mapreduce.task.attempt.id
>
> What causes this warning? I don't use mapred.task.id anywhere in the
> configurations...

I think it's used internally, so there's nothing to worry about.

Tom

>
> Felix Halim
>

I have created a new candidate build for Hadoop 0.21.0. This fixes
MAPREDUCE-1920 and MAPREDUCE-1926 which were found for the previous
release candidate (0).

This release is being classified as a minor release, which means that
it is API compatible with 0.20.2.

*** This release candidate has not been tested extensively, so it
should not be considered stable.
*** Please download, test and vote before the vote closes on Wednesday
18 August.

http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-1/

The hadoop-0.21.0.tar.gz file is an old-style combined release which
includes Common, HDFS, and MapReduce.

Thanks,
Tom