I've created another release candidate for hadoop-0.23.1 that I would like to release.

It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc1/
The hadoop-0.23.1-rc1 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc1
The maven artifacts for hadoop-0.23.1-rc1 are also available at repository.apache.org.

Since hadoop-0.23.1-rc0 last week, we've fixed some important bugs:
# HDFS-2923. Namenode IPC handler count uses the wrong configuration key.
# MAPREDUCE-3858. Task attempt failure during commit results in task never completing.
# MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then the recovery.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

I built a tar from the tag, ran some basic tests on a single node cluster and it ran just fine.  I am a bit concerned about the unit tests though.  I ran all of the unit tests and I saw several failures

org.apache.hadoop.hdfs.server.namenode.TestStartup
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
org.apache.hadoop.mapred.TestLazyOutput
 org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat

The only JIRA I could find for these failures is MAPREDUCE-3776 which covers TestRMContainerAllocator.

I am not sure if 0.23.1 is being considered alpha quality and if so do we want to let unit test failures block the release.

--Bobby Evans

On 2/14/12 11:28 PM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:

I've created another release candidate for hadoop-0.23.1 that I would like to release.

It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc1/
The hadoop-0.23.1-rc1 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc1
The maven artifacts for hadoop-0.23.1-rc1 are also available at repository.apache.org.

Since hadoop-0.23.1-rc0 last week, we've fixed some important bugs:
# HDFS-2923. Namenode IPC handler count uses the wrong configuration key.
# MAPREDUCE-3858. Task attempt failure during commit results in task never completing.
# MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then the recovery.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

I checked this RC, based on my check-list:

 - Signatures and message digests all are good.
 - The top level full LICENSE, NOTICE and README for both the source and
binary artifacts are good
 - CHANGES.txt for common, hdfs and mapped are correctly located.
 - Able to build the tars out of the source tar ball.

Testing: All testing on single node, unsecured, default mode.
  - Started HDFS daemons successfully , created directories.
  - Successfully started YARN daemons - ResourceManager and NodeManager
  - Successfully started mapred history server.
  - Ran wordcount, pi, random writer, sort, grep and they all pass just
fine.

Even before the RC, been actively doing performance benchmarking and
testing on big clusters with this code. 0.23.1 clearly has much more
improvements compared to 0.23., even though it is still (the next level)
alpha.

With that, here's my +1 for the latest RC.

Thanks,
+Vinod

On Tue, Feb 14, 2012 at 9:28 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> I've created another release candidate for hadoop-0.23.1 that I would like
> to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc1/
> The hadoop-0.23.1-rc1 svn tag:
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc1
> The maven artifacts for hadoop-0.23.1-rc1 are also available at
> repository.apache.org.
>
> Since hadoop-0.23.1-rc0 last week, we've fixed some important bugs:
> # HDFS-2923. Namenode IPC handler count uses the wrong configuration key.
> # MAPREDUCE-3858. Task attempt failure during commit results in task never
> completing.
> # MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and
> then the recovery.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

I would like to see HADOOP-8054 put into 0.23.1.  This is a regression that breaks hive.

--Bobby Evans

On 2/14/12 11:28 PM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:

I've created another release candidate for hadoop-0.23.1 that I would like to release.

It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc1/
The hadoop-0.23.1-rc1 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc1
The maven artifacts for hadoop-0.23.1-rc1 are also available at repository.apache.org.

Since hadoop-0.23.1-rc0 last week, we've fixed some important bugs:
# HDFS-2923. Namenode IPC handler count uses the wrong configuration key.
# MAPREDUCE-3858. Task attempt failure during commit results in task never completing.
# MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then the recovery.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Makes sense, let me spin rc2.

On Feb 17, 2012, at 8:20 AM, Robert Evans wrote:

> I would like to see HADOOP-8054 put into 0.23.1.  This is a regression that breaks hive.
>
> --Bobby Evans
>
> On 2/14/12 11:28 PM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:
>
> I've created another release candidate for hadoop-0.23.1 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc1/
> The hadoop-0.23.1-rc1 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc1
> The maven artifacts for hadoop-0.23.1-rc1 are also available at repository.apache.org.
>
> Since hadoop-0.23.1-rc0 last week, we've fixed some important bugs:
> # HDFS-2923. Namenode IPC handler count uses the wrong configuration key.
> # MAPREDUCE-3858. Task attempt failure during commit results in task never completing.
> # MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then the recovery.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

What about MAPREDUCE-3685 which reportedly boosts performance for
MergeManager ?
Should it be considered for next RC ?
Cheers
On Sat, Feb 18, 2012 at 9:31 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> Makes sense, let me spin rc2.
>
> On Feb 17, 2012, at 8:20 AM, Robert Evans wrote:
>
> > I would like to see HADOOP-8054 put into 0.23.1.  This is a regression
> that breaks hive.
> >
> > --Bobby Evans
> >
> > On 2/14/12 11:28 PM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:
> >
> > I've created another release candidate for hadoop-0.23.1 that I would
> like to release.
> >
> > It is available at:
> http://people.apache.org/~acmurthy/hadoop-0.23.1-rc1/
> > The hadoop-0.23.1-rc1 svn tag:
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc1
> > The maven artifacts for hadoop-0.23.1-rc1 are also available at
> repository.apache.org.
> >
> > Since hadoop-0.23.1-rc0 last week, we've fixed some important bugs:
> > # HDFS-2923. Namenode IPC handler count uses the wrong configuration key.
> > # MAPREDUCE-3858. Task attempt failure during commit results in task
> never completing.
> > # MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and
> then the recovery.
> >
> > Please try the release and vote; the vote will run for the usual 7 days.
> >
> > thanks,
> > Arun
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

I downloaded and checked this rc, some of the things I checked:

  - release notes are present both binary and src
  - html documentation present in binary: share/doc/hadoop/index.html
  - release is present at:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoo
p-common/0.23.1/
  - tag looks good and matches src tarball contents
  - LICENSE, NOTICE and REAEDME are good
  - md5 and signatures good
  - src tarball builds

Testing: All testing on single node, unsecured
  - all daemons started - hdfs/yarn/history server
  - Ran wordcount, sleep, teragen, terasort, teravalidate and all passed
  
Questions (I'm new to this process):
  - in binary package container-executor appears to be 64 bit, but the
libraries (libhadoop.a, etc) are 32 bit - what is the binary supposed to
support - 32 or 64?
  - CHANGES.txt contains the date as (2012-02-08), does this need to be
updated for the new date of the rc1 vote?
Thanks,
Tom
On 2/14/12 11:28 PM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:

> I've created another release candidate for hadoop-0.23.1 that I would like to
> release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc1/
> The hadoop-0.23.1-rc1 svn tag:
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc1
> The maven artifacts for hadoop-0.23.1-rc1 are also available at
> repository.apache.org.
>
> Since hadoop-0.23.1-rc0 last week, we've fixed some important bugs:
> # HDFS-2923. Namenode IPC handler count uses the wrong configuration key.
> # MAPREDUCE-3858. Task attempt failure during commit results in task never
> completing.
> # MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then
> the recovery.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

l
2012. 2. 15. 오후 2:29에 "Arun C Murthy" <[EMAIL PROTECTED]>님이 작성:

> I've created another release candidate for hadoop-0.23.1 that I would like
> to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc1/
> The hadoop-0.23.1-rc1 svn tag:
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc1
> The maven artifacts for hadoop-0.23.1-rc1 are also available at
> repository.apache.org.
>
> Since hadoop-0.23.1-rc0 last week, we've fixed some important bugs:
> # HDFS-2923. Namenode IPC handler count uses the wrong configuration key.
> # MAPREDUCE-3858. Task attempt failure during commit results in task never
> completing.
> # MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and
> then the recovery.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

I've created another release candidate for hadoop-0.23.1 that I would like to release.

It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

+1

I built it from source, and ran a job on a single node cluster.  I did not run any of the unit tests as this is still alpha quality.

--Bobby Evans

On 2/18/12 1:23 AM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:

I've created another release candidate for hadoop-0.23.1 that I would like to release.

It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

+1

I downloaded and checked this rc, some of the things I checked:

  - release notes are present both binary and src
  - html documentation present in binary: share/doc/hadoop/index.html
  - release is present at:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoo
p-common/0.23.1/
  - LICENSE, NOTICE and README are good
  - md5 and signatures good
  - src tarball builds
  - in binary package container-executor and libraries (libhadoop.a, etc)
are 32 bit
  - CHANGES.txt files look good

Testing: All testing on single node, unsecured
  - all daemons started - hdfs/yarn/history server
  - Ran wordcount, sleep, teragen, terasort, teravalidate and all passed
  
Thanks,
Tom

On 2/18/12 1:23 AM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:

> I've created another release candidate for hadoop-0.23.1 that I would like to
> release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
> The hadoop-0.23.1-rc2 svn tag:
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
> The maven artifacts for hadoop-0.23.1-rc2 are also available at
> repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

+1.
 Downloaded the release. Set up a single node non secure cluster. Ran
a couple of example jobs on it. Looking forward to getting it out to
our users.

Mahadev Konar
Hortonworks Inc.
http://hortonworks.com/

On Tue, Feb 21, 2012 at 5:11 PM, Thomas Graves <[EMAIL PROTECTED]> wrote:
> +1
>
> I downloaded and checked this rc, some of the things I checked:
>
>  - release notes are present both binary and src
>  - html documentation present in binary: share/doc/hadoop/index.html
>  - release is present at:
> https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoo
> p-common/0.23.1/
>  - LICENSE, NOTICE and README are good
>  - md5 and signatures good
>  - src tarball builds
>  - in binary package container-executor and libraries (libhadoop.a, etc)
> are 32 bit
>  - CHANGES.txt files look good
>
> Testing: All testing on single node, unsecured
>  - all daemons started - hdfs/yarn/history server
>  - Ran wordcount, sleep, teragen, terasort, teravalidate and all passed
>
> Thanks,
> Tom
>
> On 2/18/12 1:23 AM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:
>
>> I've created another release candidate for hadoop-0.23.1 that I would like to
>> release.
>>
>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
>> The hadoop-0.23.1-rc2 svn tag:
>> https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
>> The maven artifacts for hadoop-0.23.1-rc2 are also available at
>> repository.apache.org.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>> thanks,
>> Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>

Verified the checksums.
Built from source, deployed a single node non-secure cluster. Ran several
jobs - with the Fifo and CapacityScheduler which completed successfully.

+1

On Fri, Feb 17, 2012 at 11:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> I've created another release candidate for hadoop-0.23.1 that I would like
> to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
> The hadoop-0.23.1-rc2 svn tag:
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
> The maven artifacts for hadoop-0.23.1-rc2 are also available at
> repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

I checked this RC, based on the same check-list as RC1:

 - Signatures, message digests, the top level full LICENSE.txt,
NOTICE.txt and README.txt for both the source and binary artifacts are
all good. CHANGES.txt for common, hdfs and mapped are correctly
located like before.
 - Able to build the tars out of the source tar ball.

Testing: All testing on single node, unsecured, default mode. All
under 10 minutes :)
  - Started HDFS daemons successfully , created directories.
  - Successfully started YARN daemons - ResourceManager and
NodeManager and mapred history server.
  - Ran wordcount, pi, random writer, sort, grep and they all pass.

+1 for the latest RC, it's time we push this out!!

Thanks,
+Vinod

On Fri, Feb 17, 2012 at 11:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created another release candidate for hadoop-0.23.1 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
> The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
> The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

+1

Checked signatures and checksums, also ran a few jobs successfully on
a small cluster via https://issues.apache.org/jira/browse/WHIRR-391.

Re: HDFS-2991 - how about adding a release note warning about it?

Tom

On Fri, Feb 17, 2012 at 11:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created another release candidate for hadoop-0.23.1 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
> The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
> The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

On Fri, Feb 17, 2012 at 11:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created another release candidate for hadoop-0.23.1 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
> The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
> The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.

-1 (non-binding) from Bigtop

The changes introduced over here:
     https://issues.apache.org/jira/browse/MAPREDUCE-3880

not only make it impossible to build the right version
of container executor without hand-editing the pom.xml,
but also make it impossible to deploy the secure
version of hadoop without additional burden of installing
32bit legacy support on all the slave nodes.

Thanks,
Roman.

Roman,

you are compiling on 64 bit machines?

> not only make it impossible to build the right version
> of container executor without hand-editing the pom.xml,

The idea was to have 32 bit binary all the time, can you let us know
what issue you are facing. During the build itself?

> but also make it impossible to deploy the secure
> version of hadoop without additional burden of installing
> 32bit legacy support on all the slave nodes.

Again, what do you mean by this? You mean you will have to install all
32 bit dependencies that the container-executor may have on all nodes?
AFAIU, if you have a single 32bit lib/binary on a 64 bit machine, you
don't need any other 32bit lib dependencies. Please correct me if I am
wrong. Myself cannot test it exactly like the way you are testing as I
don't have access to nodes which don't have 32bits installed, so more
specific details are helpful.

Thanks,
+Vinod

On Wed, Feb 22, 2012 at 7:45 PM, Vinod Kumar Vavilapalli
<[EMAIL PROTECTED]> wrote:
> Roman,
>
> you are compiling on 64 bit machines?

I'm compiling on both 32bit and 64bit architectures.

>> not only make it impossible to build the right version
>> of container executor without hand-editing the pom.xml,
>
> The idea was to have 32 bit binary all the time

This is a flawed idea and that's one half of the reason I gave this RC
a -1. You can NOT force all 64bit environments to always have
a 32bit set of libraries, etc. Basically, with this release, all of a sudden
you force all the deployments to install quite a lot of extra RPMs/DEBs
for no other reason but to workaround a deficiency in the default Hadoop
build. Now, that would been enough to give this RC a -0 but
an added issue of prohibiting regeneration of the correct packages with the
reasonable set of binaries makes it -1.

> can you let us know what issue you are facing. During the build itself?

http://bigtop01.cloudera.org:8080/view/Bigtop-hadoop-0.23/job/Bigtop-hadoop-0.23-Hadoop/12/

Click on any of the distributions to see how the build fails.

Note -- these are clean VMs. We take special precautions
not to pollute them with any extra dependencies. These VMs are
what distribution providers (CentOS, SLES, Canonical) have deemed
to be the worthy of public consumption on Amazon's EC2.

>> but also make it impossible to deploy the secure
>> version of hadoop without additional burden of installing
>> 32bit legacy support on all the slave nodes.
>
> Again, what do you mean by this? You mean you will have to install all
> 32 bit dependencies that the container-executor may have on all nodes?

That is correct. These dependencies are not available by default, have
an extra layer of complexity and are being slowly phased out of support
(to the best of my knowledge).

> AFAIU, if you have a single 32bit lib/binary on a 64 bit machine, you
> don't need any other 32bit lib dependencies. Please correct me if I am
> wrong.

$ tar xzvf hadoop-0.23.1.tag.gz
$ ldd bin/container-executor
linux-gate.so.1 =>  (0xf770a000)
libc.so.6 => /lib32/libc.so.6 (0xf758d000)
/lib/ld-linux.so.2 (0xf770b000)

That /lib32/libc.so.6 dependency is a problem.

> Myself cannot test it exactly like the way you are testing as I
> don't have access to nodes which don't have 32bits installed, so more
> specific details are helpful.

You don't have to go far to get a testbed -- any default 64bit VM
of all the major Linux distributions on EC2 would suffice.

Thanks,
Roman.

-1, unfortunately. HDFS-2991 is a blocker regression introduced in
0.23.1. See the JIRA for instructions on how to reproduce on the rc2
build.

-Todd

On Fri, Feb 17, 2012 at 11:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created another release candidate for hadoop-0.23.1 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
> The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
> The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

--
Todd Lipcon
Software Engineer, Cloudera

Todd,

>From your analysis at HDFS-2991, looks like this was there in 0.23
too. Also, seems this happens only at scale, and only (paraphrasing
you) "when the file is reopened for append on an exact block
boundary".

Agree it is a critical fix, but given above, can we proceed along with
0.23.1? Anyways, 0.23.1 is still an alpha (albeit of next level), so
I'd think we can get that in for 0.23.2.

+Vinod

On Wed, Feb 22, 2012 at 6:43 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> -1, unfortunately. HDFS-2991 is a blocker regression introduced in
> 0.23.1. See the JIRA for instructions on how to reproduce on the rc2
> build.
>
> -Todd
>
> On Fri, Feb 17, 2012 at 11:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>> I've created another release candidate for hadoop-0.23.1 that I would like to release.
>>
>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
>> The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
>> The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>> thanks,
>> Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

On Wed, Feb 22, 2012 at 7:51 PM, Vinod Kumar Vavilapalli
<[EMAIL PROTECTED]> wrote:
> Todd,
>
> From your analysis at HDFS-2991, looks like this was there in 0.23
> too. Also, seems this happens only at scale, and only (paraphrasing
> you) "when the file is reopened for append on an exact block
> boundary".

Let me clarify: HDFS-2991 basically has two halves:
First half (been present "forever"): when we append() on a block
boundary, we don't log an OP_ADD
Second half (new due to HDFS-2718): if we get an OP_CLOSE for a file
we haven't OP_ADDed, we'll get a ClassCastException on startup.

So even though the first half isn't a regression, the regression in
the second half means that this longstanding bug will now actually
prevent startup.

Also, there's nothing related to scale here. I happened to run into it
doing scale tests, but it turned out to not be relevant. You'll see it
if you run TestDFSIO with standard parameters on trunk or 23.1 (that's
how I discovered it).

>
> Agree it is a critical fix, but given above, can we proceed along with
> 0.23.1? Anyways, 0.23.1 is still an alpha (albeit of next level), so
> I'd think we can get that in for 0.23.2.

Alright, consider me -0, though it's pretty nasty once you run into
it. The only way I could start my NN again without losing data was to
recompile with the fix in place.

-Todd

>
> On Wed, Feb 22, 2012 at 6:43 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>> -1, unfortunately. HDFS-2991 is a blocker regression introduced in
>> 0.23.1. See the JIRA for instructions on how to reproduce on the rc2
>> build.
>>
>> -Todd
>>
>> On Fri, Feb 17, 2012 at 11:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>>> I've created another release candidate for hadoop-0.23.1 that I would like to release.
>>>
>>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
>>> The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
>>> The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.
>>>
>>> Please try the release and vote; the vote will run for the usual 7 days.
>>>
>>> thanks,
>>> Arun
>>>
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera

--
Todd Lipcon
Software Engineer, Cloudera

On Feb 23, 2012, at 10:30 AM, Todd Lipcon wrote:
>
> Alright, consider me -0, though it's pretty nasty once you run into
> it. The only way I could start my NN again without losing data was to
> recompile with the fix in place.

Thanks Todd.

I'm thinking 23.1 has stewed long enough that we can get it out and follow up with a 23.2 in a week or so with:
a) HDFS-2991
b) The build issue for LCE that Roman pointed out.
c) A few other useful ones like MAPREDUCE-3884 and some nice to have, but safe, fixes such as MAPREDUCE-3897, MAPREDUCE-3900 etc.

OTOH, we don't have to block the PB merge either, I can just do 23.2 soon off a branch from branch-0.23.1.

Makes sense?

Arun

+1 to that. I'd really like to see 0.23.1 get out sooner than later.
0.23.1 is a huge improvement over 0.23.0 and would really like to see
folks starting to play around with it!

Mahadev Konar
Hortonworks Inc.
http://hortonworks.com/

On Wed, Feb 22, 2012 at 9:32 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>
> On Feb 23, 2012, at 10:30 AM, Todd Lipcon wrote:
>>
>> Alright, consider me -0, though it's pretty nasty once you run into
>> it. The only way I could start my NN again without losing data was to
>> recompile with the fix in place.
>
> Thanks Todd.
>
> I'm thinking 23.1 has stewed long enough that we can get it out and follow up with a 23.2 in a week or so with:
> a) HDFS-2991
> b) The build issue for LCE that Roman pointed out.
> c) A few other useful ones like MAPREDUCE-3884 and some nice to have, but safe, fixes such as MAPREDUCE-3897, MAPREDUCE-3900 etc.
>
> OTOH, we don't have to block the PB merge either, I can just do 23.2 soon off a branch from branch-0.23.1.
>
> Makes sense?
>
> Arun
>

+1.

Creating 23.2 from 23.1 is a good idea. That way we can merge protobuf
stuff into 0.23.
On Wed, Feb 22, 2012 at 9:37 PM, Mahadev Konar <[EMAIL PROTECTED]>wrote:

> +1 to that. I'd really like to see 0.23.1 get out sooner than later.
> 0.23.1 is a huge improvement over 0.23.0 and would really like to see
> folks starting to play around with it!
>
> Mahadev Konar
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> On Wed, Feb 22, 2012 at 9:32 PM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >
> > On Feb 23, 2012, at 10:30 AM, Todd Lipcon wrote:
> >>
> >> Alright, consider me -0, though it's pretty nasty once you run into
> >> it. The only way I could start my NN again without losing data was to
> >> recompile with the fix in place.
> >
> > Thanks Todd.
> >
> > I'm thinking 23.1 has stewed long enough that we can get it out and
> follow up with a 23.2 in a week or so with:
> > a) HDFS-2991
> > b) The build issue for LCE that Roman pointed out.
> > c) A few other useful ones like MAPREDUCE-3884 and some nice to have,
> but safe, fixes such as MAPREDUCE-3897, MAPREDUCE-3900 etc.
> >
> > OTOH, we don't have to block the PB merge either, I can just do 23.2
> soon off a branch from branch-0.23.1.
> >
> > Makes sense?
> >
> > Arun
> >
>

Hey Arun,

I think HADOOP-8105 is also worth another RC. It's an incompatible
change that breaks a bunch of the Hive 0.8 tests.

Thanks,
Eli

On Fri, Feb 17, 2012 at 11:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created another release candidate for hadoop-0.23.1 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
> The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
> The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

Hey Arun,

Looks like this has already been fixed, apologies for the noise!

Thanks,
Eli

On Wed, Feb 22, 2012 at 8:30 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
> Hey Arun,
>
> I think HADOOP-8105 is also worth another RC. It's an incompatible
> change that breaks a bunch of the Hive 0.8 tests.
>
> Thanks,
> Eli
>
> On Fri, Feb 17, 2012 at 11:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>> I've created another release candidate for hadoop-0.23.1 that I would like to release.
>>
>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc2/
>> The hadoop-0.23.1-rc2 svn tag: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.1-rc2
>> The maven artifacts for hadoop-0.23.1-rc2 are also available at repository.apache.org.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>> thanks,
>> Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>

Fyi - vote for 0.23.7 going on in the dev lists.  Forwarding to general as
the a few people have told me they didn't see this.

Tom
On 4/11/13 2:55 PM, "Thomas Graves" <[EMAIL PROTECTED]> wrote:

>I've created a release candidate (RC0) for hadoop-0.23.7 that I would like
>to release.
>
>This release is a sustaining release with several important bug fixes in
>it.
>
>The RC is available at:
>http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/
>The RC tag in svn is here:
>http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/
>
>The maven artifacts are available via repository.apache.org.
>
>Please try the release and vote; the vote will run for the usual 7 days.
>
>thanks,
>Tom Graves
>