Hi everybody,

We reached the homestretch for hadoop-0.22 release.
The testing proceeds very well.
- The builds are building
- HDFS and MR are running under load well now.
- We applied pretty simple custom patches to make Hive (7) and Pig (8)
work with 0.22.
- HBase (92), which is expected to be released in the near future as I
hear, is fully compatible with 0.22.

There are 4 main items remaining on my plate.

1. We are down to four blockers, which represent three issues:
HADOOP-7035. Document incompatible API changes between releases
MAPREDUCE-1100 / 1716. User's task-logs filling up local disks on the
TaskTrackers
HDFS-2498. TestParallelRead times out

I'd like also to include
HDFS-1900. Clean up config key collision between common and HDFS
and potentially further bug fixes.

2. Assemble build is needed.
By assemble build I mean a Jenkins target which produces a tarball
that includes all three sub-projects. It is like a daily build. One of
this builds is assigned to be release a candidate and then the
release. We can do it in traditional way, based on scripts developed
in HADOOP-6846. Or we can try to do it with BigTop.

3. Jetty upgrade.
0.22 branch is on Jetty 6.1.14, while everything else moved to 6.1.26.
Based on experiments I ran with Jetty 6.1.14 tasks hang forever if
things go wrong, while with Jetty 6.1.26 you get a lot of "fetch
failure"s. The latter at least fails fast, and does not need manual
intervention, so I'd go for the upgrade.

4. Documentation.
In addition to HADOOP-7035, which according to Tom needs a rerun of
the tool, we need to confirm the documentation is up to date. I think
it is, but re-checking would be good.

I seek help from the good people of Hadoopland on these last steps.
If you have suggestions for inclusion or plan to help with the items,
please reply to the thread or drop me a line. I understand the next
two weeks will be busy with conferences. I plan to release after
Apachecon by mid November. If the assemble build is ready soon people
will be able to try it early.

Thanks,
--Konstantin

Hey Konstantin,

On Fri, Oct 28, 2011 at 2:59 PM, Konstantin Shvachko
<[EMAIL PROTECTED]> wrote:
> 3. Jetty upgrade.
> 0.22 branch is on Jetty 6.1.14, while everything else moved to 6.1.26.
> Based on experiments I ran with Jetty 6.1.14 tasks hang forever if
> things go wrong, while with Jetty 6.1.26 you get a lot of "fetch
> failure"s. The latter at least fails fast, and does not need manual
> intervention, so I'd go for the upgrade.

I believe https://issues.apache.org/jira/browse/MAPREDUCE-2980 would help here.

--
Harsh J

On Fri, Oct 28, 2011 at 2:29 AM, Konstantin Shvachko
<[EMAIL PROTECTED]> wrote:
> Hi everybody,
>
> We reached the homestretch for hadoop-0.22 release.
> The testing proceeds very well.
> - The builds are building
> - HDFS and MR are running under load well now.
> - We applied pretty simple custom patches to make Hive (7) and Pig (8)
> work with 0.22.
> - HBase (92), which is expected to be released in the near future as I
> hear, is fully compatible with 0.22.

Yes, HBase is the only model citizen here. As for the rest of component
I really would like to figure out what the plan is. Given that the patches
are not big, is there any way Pig and Hive guys can be convinced
to spin a maintenance release? I can certainly help with the logistics.

Right now, Bigtop is at a point where an entire stack based on Hadoop
.22 can be deployed (via puppet) as a single step. That helps a great
deal in validating the release. The problem is, that give Bigtop policies
of not releasing with custom patches (we only package straight Apache
releases) it would be a shame not to have.22 stack available for a single
click install/deployment.

> 2. Assemble build is needed.
> By assemble build I mean a Jenkins target which produces a tarball
> that includes all three sub-projects. It is like a daily build. One of
> this builds is assigned to be release a candidate and then the
> release. We can do it in traditional way, based on scripts developed
> in HADOOP-6846. Or we can try to do it with BigTop.

I can help with that. I was playing with the .22 build assembly yesterday
and it seems that it shouldn't be a problem. I can also test it as a tarball
once it is assembled.

Thanks,
Roman.

Konstantin,

Great news! The progress is pretty spectacular I'd say.

I am slowly working on Jetty back-port. Unfortunately, other things require my
attention too. Hopefully, it will be finished over next a couple days or so.

Cos

On Fri, Oct 28, 2011 at 02:29AM, Konstantin Shvachko wrote:
> Hi everybody,
>
> We reached the homestretch for hadoop-0.22 release.
> The testing proceeds very well.
> - The builds are building
> - HDFS and MR are running under load well now.
> - We applied pretty simple custom patches to make Hive (7) and Pig (8)
> work with 0.22.
> - HBase (92), which is expected to be released in the near future as I
> hear, is fully compatible with 0.22.
>
> There are 4 main items remaining on my plate.
>
> 1. We are down to four blockers, which represent three issues:
> HADOOP-7035. Document incompatible API changes between releases
> MAPREDUCE-1100 / 1716. User's task-logs filling up local disks on the
> TaskTrackers
> HDFS-2498. TestParallelRead times out
>
> I'd like also to include
> HDFS-1900. Clean up config key collision between common and HDFS
> and potentially further bug fixes.
>
> 2. Assemble build is needed.
> By assemble build I mean a Jenkins target which produces a tarball
> that includes all three sub-projects. It is like a daily build. One of
> this builds is assigned to be release a candidate and then the
> release. We can do it in traditional way, based on scripts developed
> in HADOOP-6846. Or we can try to do it with BigTop.
>
> 3. Jetty upgrade.
> 0.22 branch is on Jetty 6.1.14, while everything else moved to 6.1.26.
> Based on experiments I ran with Jetty 6.1.14 tasks hang forever if
> things go wrong, while with Jetty 6.1.26 you get a lot of "fetch
> failure"s. The latter at least fails fast, and does not need manual
> intervention, so I'd go for the upgrade.
>
> 4. Documentation.
> In addition to HADOOP-7035, which according to Tom needs a rerun of
> the tool, we need to confirm the documentation is up to date. I think
> it is, but re-checking would be good.
>
> I seek help from the good people of Hadoopland on these last steps.
> If you have suggestions for inclusion or plan to help with the items,
> please reply to the thread or drop me a line. I understand the next
> two weeks will be busy with conferences. I plan to release after
> Apachecon by mid November. If the assemble build is ready soon people
> will be able to try it early.
>
> Thanks,
> --Konstantin

For upgrade we can directly (or indirectly) use HADOOP-7450
It's pre-mavenization.
Cos, you might want to use it.
--Konstantin
On Fri, Oct 28, 2011 at 3:26 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Hey Konstantin,
>
> On Fri, Oct 28, 2011 at 2:59 PM, Konstantin Shvachko
> <[EMAIL PROTECTED]> wrote:
>> 3. Jetty upgrade.
>> 0.22 branch is on Jetty 6.1.14, while everything else moved to 6.1.26.
>> Based on experiments I ran with Jetty 6.1.14 tasks hang forever if
>> things go wrong, while with Jetty 6.1.26 you get a lot of "fetch
>> failure"s. The latter at least fails fast, and does not need manual
>> intervention, so I'd go for the upgrade.
>
> I believe https://issues.apache.org/jira/browse/MAPREDUCE-2980 would help here.
>
> --
> Harsh J
>

Yes, this is exactly what I took as the template last week ;( Thanks.

On Fri, Oct 28, 2011 at 01:51PM, Konstantin Shvachko wrote:
> For upgrade we can directly (or indirectly) use HADOOP-7450
> It's pre-mavenization.
> Cos, you might want to use it.
> --Konstantin
>
>
> On Fri, Oct 28, 2011 at 3:26 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> > Hey Konstantin,
> >
> > On Fri, Oct 28, 2011 at 2:59 PM, Konstantin Shvachko
> > <[EMAIL PROTECTED]> wrote:
> >> 3. Jetty upgrade.
> >> 0.22 branch is on Jetty 6.1.14, while everything else moved to 6.1.26.
> >> Based on experiments I ran with Jetty 6.1.14 tasks hang forever if
> >> things go wrong, while with Jetty 6.1.26 you get a lot of "fetch
> >> failure"s. The latter at least fails fast, and does not need manual
> >> intervention, so I'd go for the upgrade.
> >
> > I believe https://issues.apache.org/jira/browse/MAPREDUCE-2980 would help here.
> >
> > --
> > Harsh J
> >

On Fri, Oct 28, 2011 at 2:29 AM, Konstantin Shvachko
<[EMAIL PROTECTED]>wrote:

> Hi everybody,
>
> We reached the homestretch for hadoop-0.22 release.
>

How are you going to deal with MAPREDUCE-2178, which we should mark as
resolved? It fixes a critical security hole in the task-controller and is
fixed in both 0.20.2xx and 0.23.

The two reasonable options:

* Remove the task-controller. (This was the fix that was used in 0.21.)

* Forward port the patch from 2xx.

-- Owen

MAPREDUCE-2767 removed LinuxTaskController as discussed earlier.
The idea is to port changes from 20.security in the subsequent release.

Thanks,
--Konstantin

On Fri, Oct 28, 2011 at 4:43 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
> On Fri, Oct 28, 2011 at 2:29 AM, Konstantin Shvachko
> <[EMAIL PROTECTED]>wrote:
>
>> Hi everybody,
>>
>> We reached the homestretch for hadoop-0.22 release.
>>
>
> How are you going to deal with MAPREDUCE-2178, which we should mark as
> resolved? It fixes a critical security hole in the task-controller and is
> fixed in both 0.20.2xx and 0.23.
>
> The two reasonable options:
>
> * Remove the task-controller. (This was the fix that was used in 0.21.)
>
> * Forward port the patch from 2xx.
>
> -- Owen
>

Konstantin,
  HADOOP-7450 (and its subs for HDFS and MR) have patches for 0.22 now.

On Fri, Oct 28, 2011 at 10:53AM, Konstantin Boudnik wrote:
> Konstantin,
>
> Great news! The progress is pretty spectacular I'd say.
>
> I am slowly working on Jetty back-port. Unfortunately, other things require my
> attention too. Hopefully, it will be finished over next a couple days or so.
>
> Cos
>
> On Fri, Oct 28, 2011 at 02:29AM, Konstantin Shvachko wrote:
> > Hi everybody,
> >
> > We reached the homestretch for hadoop-0.22 release.
> > The testing proceeds very well.
> > - The builds are building
> > - HDFS and MR are running under load well now.
> > - We applied pretty simple custom patches to make Hive (7) and Pig (8)
> > work with 0.22.
> > - HBase (92), which is expected to be released in the near future as I
> > hear, is fully compatible with 0.22.
> >
> > There are 4 main items remaining on my plate.
> >
> > 1. We are down to four blockers, which represent three issues:
> > HADOOP-7035. Document incompatible API changes between releases
> > MAPREDUCE-1100 / 1716. User's task-logs filling up local disks on the
> > TaskTrackers
> > HDFS-2498. TestParallelRead times out
> >
> > I'd like also to include
> > HDFS-1900. Clean up config key collision between common and HDFS
> > and potentially further bug fixes.
> >
> > 2. Assemble build is needed.
> > By assemble build I mean a Jenkins target which produces a tarball
> > that includes all three sub-projects. It is like a daily build. One of
> > this builds is assigned to be release a candidate and then the
> > release. We can do it in traditional way, based on scripts developed
> > in HADOOP-6846. Or we can try to do it with BigTop.
> >
> > 3. Jetty upgrade.
> > 0.22 branch is on Jetty 6.1.14, while everything else moved to 6.1.26.
> > Based on experiments I ran with Jetty 6.1.14 tasks hang forever if
> > things go wrong, while with Jetty 6.1.26 you get a lot of "fetch
> > failure"s. The latter at least fails fast, and does not need manual
> > intervention, so I'd go for the upgrade.
> >
> > 4. Documentation.
> > In addition to HADOOP-7035, which according to Tom needs a rerun of
> > the tool, we need to confirm the documentation is up to date. I think
> > it is, but re-checking would be good.
> >
> > I seek help from the good people of Hadoopland on these last steps.
> > If you have suggestions for inclusion or plan to help with the items,
> > please reply to the thread or drop me a line. I understand the next
> > two weeks will be busy with conferences. I plan to release after
> > Apachecon by mid November. If the assemble build is ready soon people
> > will be able to try it early.
> >
> > Thanks,
> > --Konstantin

On Fri, Oct 28, 2011 at 2:29 AM, Konstantin Shvachko
<[EMAIL PROTECTED]> wrote:
> 2. Assemble build is needed.
> By assemble build I mean a Jenkins target which produces a tarball
> that includes all three sub-projects. It is like a daily build. One of
> this builds is assigned to be release a candidate and then the
> release. We can do it in traditional way, based on scripts developed
> in HADOOP-6846. Or we can try to do it with BigTop.

For anybody who's interested in testing the potential release and HBase
0.92 RC here's what I've created on top of Bigtop:
  1. For a more classical tarball approach to testing/deployment, there's
      an assembly build Apache jenkins job:
          https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-22-Build/
     A potential release artifacts could be pulled from there (along with the
     usual md5/sha1 info).

  2. For a more convenient installation experience of Hadoop + HBase, here's
  a Bigtop build of package artifacts:
      CentOS/RHEL:
http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/job/Bigtop-hadoop22-centos5/
      Ubuntu: http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/job/Bigtop-hadoop22-ubuntu-lucid/
   Anybody who's interested in trying out the combination can simply grab a repo
   file from those jobs and yum/apt-get install the bits. E.g.:
      centos$ sudo wget -O /etc/yum.repos.d/bigtop.repo
http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/job/Bigtop-hadoop22-centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
      centos$ sudo yum search hadoop
      centos$ sudo yum install hadoop-conf-pseudo hadoop-hbase*
   Or:
      ubuntu$ sudo wget -O /etc/apt/sources.list.d/bigtop.list
http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/job/Bigtop-hadoop22-ubuntu-lucid/lastSuccessfulBuild/artifact/output/bigtop.list
      ubuntu$ sudo apt-get update
      ubuntu$ sudo apt-get install hadoop-conf-pseudo hadoop-hbase*

Happy testing and let me know if anything seems to be missing.

Thanks,
Roman.

Konstantin,

On Oct 28, 2011, at 2:29 AM, Konstantin Shvachko wrote:

> We reached the homestretch for hadoop-0.22 release.
>
> There are 4 main items remaining on my plate.
>
> 1. We are down to four blockers, which represent three issues:
> HADOOP-7035. Document incompatible API changes between releases
> MAPREDUCE-1100 / 1716. User's task-logs filling up local disks on the
> TaskTrackers
> HDFS-2498. TestParallelRead times out
>
...

> I seek help from the good people of Hadoopland on these last steps.
> If you have suggestions for inclusion or plan to help with the items,
> please reply to the thread or drop me a line. I understand the next
> two weeks will be busy with conferences. I plan to release after
> Apachecon by mid November. If the assemble build is ready soon people
> will be able to try it early.

I was looking through some issues and have some suggestions for inclusion (primarily MR) which should help 0.22, for your consideration.

I've created a label in jira which should help track them, please take a look:
http://s.apache.org/critical-0.22.0

thanks,
Arun

Great list Arun. I agree with most of them and will ad more.
Thanks,
--Konstantin

On Thu, Nov 24, 2011 at 2:24 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> Konstantin,
>
> On Oct 28, 2011, at 2:29 AM, Konstantin Shvachko wrote:
>
>> We reached the homestretch for hadoop-0.22 release.
>>
>> There are 4 main items remaining on my plate.
>>
>> 1. We are down to four blockers, which represent three issues:
>> HADOOP-7035. Document incompatible API changes between releases
>> MAPREDUCE-1100 / 1716. User's task-logs filling up local disks on the
>> TaskTrackers
>> HDFS-2498. TestParallelRead times out
>>
> ...
>
>> I seek help from the good people of Hadoopland on these last steps.
>> If you have suggestions for inclusion or plan to help with the items,
>> please reply to the thread or drop me a line. I understand the next
>> two weeks will be busy with conferences. I plan to release after
>> Apachecon by mid November. If the assemble build is ready soon people
>> will be able to try it early.
>
> I was looking through some issues and have some suggestions for inclusion (primarily MR) which should help 0.22, for your consideration.
>
> I've created a label in jira which should help track them, please take a look:
> http://s.apache.org/critical-0.22.0
>
> thanks,
> Arun
>
>