I've created a release candidate for hadoop-0.23.1 that I would like to release.

It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/

Some highlights:
# Since hadoop-0.23.0 in November there has been significant progress in branch-0.23 with nearly 400 jiras committed to it (68 in Common, 78 in HDFS and 242 in MapReduce).
# An important aspect is that we've done a lot of performance related work and hadoop-0.23.1 matches or exceeds performance of hadoop-1 in pretty much every aspect of HDFS & MapReduce.
# Also, several downstream projects (HBase, Pig, Oozie, Hive etc.)  seem to be playing nicely with hadoop-0.23.1.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

On Wed, Feb 8, 2012 at 1:33 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created a release candidate for hadoop-0.23.1 that I would like to release.

Not sure whether security is in scope for 0.23.1 but the following
issue is consistently
preventing me from running a kerberized cluster:
    https://issues.apache.org/jira/browse/MAPREDUCE-3728

I guess if this doesn't get fixed/worked around we at least need to
rel. note it to
safe some major frustration for the the potential users.

Thanks,
Roman.

On Wed, Feb 8, 2012 at 1:33 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> # Also, several downstream projects (HBase, Pig, Oozie, Hive etc.)  seem to be playing nicely with hadoop-0.23.1.

On that note, it would be really nice if we can, somehow, influence
the only community that
hasn't really considered Hadoop 0.23 yet -- Mahout:
     https://issues.apache.org/jira/browse/MAHOUT-822

Here at Bigtop, we'd love to offer a Bigtop 0.4.0 (or there abouts) as
the first 100% Apache
full stack of Bigdata management software based on Hadoop 0.23 line
and I think we're
getting pretty close.

Thanks,
Roman.

On Feb 8, 2012, at 8:24 AM, Roman Shaposhnik wrote:

> On Wed, Feb 8, 2012 at 1:33 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>> I've created a release candidate for hadoop-0.23.1 that I would like to release.
>
> Not sure whether security is in scope for 0.23.1 but the following
> issue is consistently
> preventing me from running a kerberized cluster:
>    https://issues.apache.org/jira/browse/MAPREDUCE-3728

Let's continue discussion on jira, we can help debug your environment.

We've run with security on for a long while now, all performance validation was with it too and we've run, literally, tens of thousands of jobs with it.

thanks,
Arun

On Wed, Feb 8, 2012 at 10:56 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> Let's continue discussion on jira, we can help debug your environment.

It could very well be an environmental issue, no disagreement there. That said,
given that we've ran into this issue right away suggests that *at least* it
needs to be understood and FAQed.

> We've run with security on for a long while now, all performance validation was with
> it too and we've run, literally, tens of thousands of jobs with it.

(this belongs to a different thread, apologies)

I also do have a work around for it, which is, at the same time incredibly easy
and incredibly insecure -- make all users share a group with yarn.

So yes, it is possible to construct and environment that unlocks me in
KRB testing,
but I wouldn't dare run something like that in production.

If you have a production-level advise on what to do -- please chime in
on a JIRA.

Thanks,
Roman.

On Wed, Feb 8, 2012 at 1:33 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created a release candidate for hadoop-0.23.1 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/

Pulled this RC into Bigtop, built on all the 6 supported platforms
(Hadoop packages
are available from here):
    http://bigtop01.cloudera.org:8080/job/Bigtop-hadoop-0.23-Hadoop/label={centos5,centos6,fedora15,fedora16,sles11,lucid}/
E.g.:
    http://bigtop01.cloudera.org:8080/job/Bigtop-hadoop-0.23-Hadoop/label=centos5/

Now running tests with the full stack (both secure and non-secure
env.). So far so good!

One minor, cosmetic thing to notice is that it would be really nice to emphasize
source tarballs over the binary one. To that end, would it be possible to call
the release tarball:
     hadoop-0.23.1.tar.gz
and the binary one:
     hadoop-0.23.1-bin.tar.gz
?

That would be more in line with most all other Apache projects I've seen doing
releases of convenience artifacts.

Thanks,
Roman.

I just committed HDFS-2923 to branch-0.23. This bug can cause big
performance issues on the NN since the number of IPC handlers will
default way too low and won't be changed with the expected config.

Since there's a workaround, it's not a regression since 0.23.0, and
0.23.1 is still going to be labeled alpha/beta, it's up to you whether
you want to spin an rc1 on account of just this bug. If there are
other issues, though, definitely worth including this in rc1.

-Todd

On Wed, Feb 8, 2012 at 1:33 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> I've created a release candidate for hadoop-0.23.1 that I would like to release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/
>
> Some highlights:
> # Since hadoop-0.23.0 in November there has been significant progress in branch-0.23 with nearly 400 jiras committed to it (68 in Common, 78 in HDFS and 242 in MapReduce).
> # An important aspect is that we've done a lot of performance related work and hadoop-0.23.1 matches or exceeds performance of hadoop-1 in pretty much every aspect of HDFS & MapReduce.
> # Also, several downstream projects (HBase, Pig, Oozie, Hive etc.)  seem to be playing nicely with hadoop-0.23.1.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

--
Todd Lipcon
Software Engineer, Cloudera

On Feb 9, 2012, at 12:05 AM, Todd Lipcon wrote:

> I just committed HDFS-2923 to branch-0.23. This bug can cause big
> performance issues on the NN since the number of IPC handlers will
> default way too low and won't be changed with the expected config.
>
> Since there's a workaround, it's not a regression since 0.23.0, and
> 0.23.1 is still going to be labeled alpha/beta, it's up to you whether
> you want to spin an rc1 on account of just this bug. If there are
> other issues, though, definitely worth including this in rc1.
>

Thanks for the heads up Todd.

I agree, HDFS-2923 seems like a good reason to do an rc1...

Arun

> -Todd
>
> On Wed, Feb 8, 2012 at 1:33 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>> I've created a release candidate for hadoop-0.23.1 that I would like to release.
>>
>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/
>>
>> Some highlights:
>> # Since hadoop-0.23.0 in November there has been significant progress in branch-0.23 with nearly 400 jiras committed to it (68 in Common, 78 in HDFS and 242 in MapReduce).
>> # An important aspect is that we've done a lot of performance related work and hadoop-0.23.1 matches or exceeds performance of hadoop-1 in pretty much every aspect of HDFS & MapReduce.
>> # Also, several downstream projects (HBase, Pig, Oozie, Hive etc.)  seem to be playing nicely with hadoop-0.23.1.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>> thanks,
>> Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Hey Arun,

I think we should consider MR-3844 for 23.1.  It's a simple fix, and w/o it
any Hive join query running on MR2 that has table size > block size will
fail.

Thanks,
Eli
On Wed, Feb 8, 2012 at 1:33 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> I've created a release candidate for hadoop-0.23.1 that I would like to
> release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/
>
> Some highlights:
> # Since hadoop-0.23.0 in November there has been significant progress in
> branch-0.23 with nearly 400 jiras committed to it (68 in Common, 78 in HDFS
> and 242 in MapReduce).
> # An important aspect is that we've done a lot of performance related work
> and hadoop-0.23.1 matches or exceeds performance of hadoop-1 in pretty much
> every aspect of HDFS & MapReduce.
> # Also, several downstream projects (HBase, Pig, Oozie, Hive etc.)  seem
> to be playing nicely with hadoop-0.23.1.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

+1. I've reviewed the patch, it needs some more work, hopefully we can get it in soon.

thanks,
Arun

On Feb 9, 2012, at 3:16 PM, Eli Collins wrote:

> Hey Arun,
>
> I think we should consider MR-3844 for 23.1.  It's a simple fix, and w/o it
> any Hive join query running on MR2 that has table size > block size will
> fail.
>
> Thanks,
> Eli
>
>
> On Wed, Feb 8, 2012 at 1:33 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>
>> I've created a release candidate for hadoop-0.23.1 that I would like to
>> release.
>>
>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/
>>
>> Some highlights:
>> # Since hadoop-0.23.0 in November there has been significant progress in
>> branch-0.23 with nearly 400 jiras committed to it (68 in Common, 78 in HDFS
>> and 242 in MapReduce).
>> # An important aspect is that we've done a lot of performance related work
>> and hadoop-0.23.1 matches or exceeds performance of hadoop-1 in pretty much
>> every aspect of HDFS & MapReduce.
>> # Also, several downstream projects (HBase, Pig, Oozie, Hive etc.)  seem
>> to be playing nicely with hadoop-0.23.1.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>> thanks,
>> Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Downloaded the tarball. Compiled Hive against 0.23.1. Could run simple queries in local derby setup successfully. But, I'm hitting https://issues.apache.org/jira/browse/HADOOP-8054 when running unit tests against 0.23.1. Did not see this problem with 0.23.0 or with earlier 0.23.1-SNAPSHOT(built on Jan 23rd).  Investigating further.

Thanks
Amareshwari

On 2/10/12 6:13 AM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:

+1. I've reviewed the patch, it needs some more work, hopefully we can get it in soon.

thanks,
Arun

On Feb 9, 2012, at 3:16 PM, Eli Collins wrote:

> Hey Arun,
>
> I think we should consider MR-3844 for 23.1.  It's a simple fix, and w/o it
> any Hive join query running on MR2 that has table size > block size will
> fail.
>
> Thanks,
> Eli
>
>
> On Wed, Feb 8, 2012 at 1:33 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>
>> I've created a release candidate for hadoop-0.23.1 that I would like to
>> release.
>>
>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/
>>
>> Some highlights:
>> # Since hadoop-0.23.0 in November there has been significant progress in
>> branch-0.23 with nearly 400 jiras committed to it (68 in Common, 78 in HDFS
>> and 242 in MapReduce).
>> # An important aspect is that we've done a lot of performance related work
>> and hadoop-0.23.1 matches or exceeds performance of hadoop-1 in pretty much
>> every aspect of HDFS & MapReduce.
>> # Also, several downstream projects (HBase, Pig, Oozie, Hive etc.)  seem
>> to be playing nicely with hadoop-0.23.1.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>> thanks,
>> Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/