Hi all,
I propose to make a 0.20.205.1 candidate soon, with the following sets of
patches:

   - deficiencies in HBase support, pointed out by the HBase team and others
   - deficiencies in webhdfs support on secure clusters
   - a couple last-minute fixes submitted to branch-0.20-security-205 that
   were too late to be included in 205.0

If you would like other patches included, and you feel it is appropriate to
have them in 205.1 rather than waiting for 206.0, please declare them by
setting the "Target Versions" field in their Jiras, and they will receive
due consideration, assuming that the proposed patch is actually
contributed, tested, reviewed, approved, and committed
to branch-0.20-security-205 by the freeze date :-)

I would like to make the rc0 candidate next Friday, so I propose to declare
205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem for
anyone, please let me know.

Thank you, and best regards,
--Matt (Release Manager)

Matt, this one is biting me on the *** in 0.20.205, but I'm not seeing how to set the target version, perhaps because it is fixed in 0.22.

https://issues.apache.org/jira/browse/HADOOP-6453

Seems like an easy low-risk fix that's been out there for a few years. What is the best way to flag this for review?

Thanks,
    - Tim.

________________________________________
From: [EMAIL PROTECTED] [[EMAIL PROTECTED]] On Behalf Of Matt Foley [[EMAIL PROTECTED]]
Sent: Friday, November 04, 2011 5:56 PM
To: [EMAIL PROTECTED]
Subject: [ANNOUNCE] Intend to build a 0.20.205.1 candidate next Friday 11 Nov.

Hi all,
I propose to make a 0.20.205.1 candidate soon, with the following sets of
patches:

   - deficiencies in HBase support, pointed out by the HBase team and others
   - deficiencies in webhdfs support on secure clusters
   - a couple last-minute fixes submitted to branch-0.20-security-205 that
   were too late to be included in 205.0

If you would like other patches included, and you feel it is appropriate to
have them in 205.1 rather than waiting for 206.0, please declare them by
setting the "Target Versions" field in their Jiras, and they will receive
due consideration, assuming that the proposed patch is actually
contributed, tested, reviewed, approved, and committed
to branch-0.20-security-205 by the freeze date :-)

I would like to make the rc0 candidate next Friday, so I propose to declare
205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem for
anyone, please let me know.

Thank you, and best regards,
--Matt (Release Manager)

The information and any attached documents contained in this message
may be confidential and/or legally privileged.  The message is
intended solely for the addressee(s).  If you are not the intended
recipient, you are hereby notified that any use, dissemination, or
reproduction is strictly prohibited and may be unlawful.  If you are
not the intended recipient, please contact the sender immediately by
return e-mail and destroy all copies of the original message.

Yes, please see request in the Jira.
Thanks,
--Matt

On Mon, Nov 7, 2011 at 5:39 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:

> And here's another one that I would like to nominate based on Bigtop
> testing:
>   https://issues.apache.org/jira/browse/MAPREDUCE-3374
>
> Any chance of pulling this into the release?
>
> Thanks,
> Roman.
>

Hi all,
*So far, the following patches have been committed and accepted for
inclusion in 0.20.205.1:*

   *Key* *Assignee* *Reporter* *Summary*
HADOOP-5124<https://issues.apache.org/jira/browse/HADOOP-5124> Hairong
Kuang Hairong Kuang A few optimizations to FsNamesystem#RecentInvalidateSets
HADOOP-6840 <https://issues.apache.org/jira/browse/HADOOP-6840> Jitendra
Nath Pandey Nicolas Spiegelberg Support non-recursive create() in
FileSystem & SequenceFile.Writer
HADOOP-6886<https://issues.apache.org/jira/browse/HADOOP-6886>
Unassigned Nicolas Spiegelberg LocalFileSystem Needs createNonRecursive API
HADOOP-7664 <https://issues.apache.org/jira/browse/HADOOP-7664> Ravi
Prakash Ravi
Prakash o.a.h.conf.Configuration complains of overriding final parameter
even if the value with which its attempting to override is the same.
HADOOP-7711 <https://issues.apache.org/jira/browse/HADOOP-7711> Arpit
Gupta Arpit
Gupta hadoop-env.sh generated from templates has duplicate info
HADOOP-7715<https://issues.apache.org/jira/browse/HADOOP-7715> Eric
Yang Arpit Gupta see log4j Error when running mr jobs and certain dfs calls
HADOOP-7728 <https://issues.apache.org/jira/browse/HADOOP-7728> Ramya
Sunil Ramya
Sunil hadoop-setup-conf.sh should be modified to enable task memory manager
HADOOP-7740 <https://issues.apache.org/jira/browse/HADOOP-7740> Arpit
Gupta Arpit
Gupta security audit logger is not on by default, fix the log4j properties
to enable the logger
HADOOP-7765<https://issues.apache.org/jira/browse/HADOOP-7765> Eric
Yang Eric Yang Debian package contain both system and tar ball layout
HADOOP-7784 <https://issues.apache.org/jira/browse/HADOOP-7784> Eric Yang Arpit
Gupta secure datanodes fail to come up stating jsvc not found
HDFS-611<https://issues.apache.org/jira/browse/HDFS-611> Zheng
Shao dhruba borthakur Heartbeats times from Datanodes increase when there
are plenty of blocks to delete
HDFS-617<https://issues.apache.org/jira/browse/HDFS-617> Kan
Zhang Kan Zhang Support for non-recursive create() in HDFS
HDFS-1257<https://issues.apache.org/jira/browse/HDFS-1257> Eric
Payne Ramkumar Vadali Race condition on FSNamesystem#recentInvalidateSets
introduced by HADOOP-5124
HDFS-1943<https://issues.apache.org/jira/browse/HDFS-1943> Matt
Foley Wei Yongjun fail to start datanode while start-dfs.sh is executed by
root user  HDFS-2065 <https://issues.apache.org/jira/browse/HDFS-2065> Uma
Maheswara Rao G Bharath Mundlapudi Fix NPE in DFSClient.getFileChecksum
HDFS-2316 <https://issues.apache.org/jira/browse/HDFS-2316> Tsz Wo
(Nicholas), SZE Tsz Wo (Nicholas), SZE webhdfs: a complete FileSystem
implementation for accessing HDFS over HTTP [[and sub-tasks]]
HDFS-2450<https://issues.apache.org/jira/browse/HDFS-2450> Daryn
Sharp Rajit Saha Only complete hostname is supported to access data via
hdfs://  MAPREDUCE-3374<https://issues.apache.org/jira/browse/MAPREDUCE-3374>
Unassigned Roman Shaposhnik src/c++/task-controller/configure is not set
executable in the tarball and that prevents task-controller from rebuilding

*The following Jiras are targetted for 0.20.205.1, but have not yet been
committed:*

   *Key* *Assignee* *Reporter* *Summary*
HADOOP-6453<https://issues.apache.org/jira/browse/HADOOP-6453> Matt
Foley Chad Metcalf Hadoop wrapper script shouldn't ignore an existing
JAVA_LIBRARY_PATH
HADOOP-7732<https://issues.apache.org/jira/browse/HADOOP-7732>
Unassigned Arpit Gupta hadoop java docs missing hdfs package
HADOOP-7751<https://issues.apache.org/jira/browse/HADOOP-7751> Eric
Yang Arpit Gupta rpm version is not being picked from the -Dversion option
in 205  HADOOP-7804 <https://issues.apache.org/jira/browse/HADOOP-7804> Arpit
Gupta Arpit Gupta enable hadoop config generator to set
dfs.block.local-path-access.user to enable short circuit read
HADOOP-7809<https://issues.apache.org/jira/browse/HADOOP-7809> Matt
Foley Joydeep Sen Sarma Backport HADOOP-5839 to 0.20-security - fixes to
ec2 scripts to allow remote job submission
HDFS-2246<https://issues.apache.org/jira/browse/HDFS-2246>
Unassigned Sanjay Radia Shortcut a local client reads to a Datanodes files
directly  HDFS-2433 <https://issues.apache.org/jira/browse/HDFS-2433>
Unassigned Robert Joseph Evans TestFileAppend4 fails intermittently
MAPREDUCE-3343 <https://issues.apache.org/jira/browse/MAPREDUCE-3343>
Unassigned Ahmed Radwan TaskTracker Out of Memory because of distributed
cache

*Please review the items you are interested in, and complete all items by
Friday for the rc0 build.*

Also, two unit tests are failing in the nightly builds.  They passed in
Build #70 (Nov 4, 2011 12:47:00 AM GMT).
Please take a look if you committed to 205 branch after that time:
    org.apache.hadoop.streaming.TestUlimit.testCommandLine<https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-0.20.205-Build/76/testReport/org.apache.hadoop.streaming/TestUlimit/testCommandLine/>

    org.apache.hadoop.mapred.TestJvmReuse.testTaskLogs<https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-0.20.205-Build/76/testReport/org.apache.hadoop.mapred/TestJvmReuse/testTaskLogs/>
Thank you,
On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:

Hey Matt,

Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
non-trivial it seems better suited for 206 than a point release. Also,
per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
"Only functionality already committed to trunk should be submitted to
a sustaining release." and this functionality does not yet have a
patch for trunk yet (let alone committed).

Thanks,
Eli

On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> Hi all,
> I propose to make a 0.20.205.1 candidate soon, with the following sets of
> patches:
>
>   - deficiencies in HBase support, pointed out by the HBase team and others
>   - deficiencies in webhdfs support on secure clusters
>   - a couple last-minute fixes submitted to branch-0.20-security-205 that
>   were too late to be included in 205.0
>
> If you would like other patches included, and you feel it is appropriate to
> have them in 205.1 rather than waiting for 206.0, please declare them by
> setting the "Target Versions" field in their Jiras, and they will receive
> due consideration, assuming that the proposed patch is actually
> contributed, tested, reviewed, approved, and committed
> to branch-0.20-security-205 by the freeze date :-)
>
> I would like to make the rc0 candidate next Friday, so I propose to declare
> 205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem for
> anyone, please let me know.
>
> Thank you, and best regards,
> --Matt (Release Manager)
>

Hi Eli,

There is no new functionality added by HDFS-2246. It is a "performance"
fix.

But I agree that it is not a trivial fix. One proposal coud be to commit
this patch to trunk too and then continue to work on HDFS-347 to make a
better fix to this problem. The godo part is that there is no API change
if/when we move to HDFS-347.

-dhruba

On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]> wrote:

> Hey Matt,
>
> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
> non-trivial it seems better suited for 206 than a point release. Also,
> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
> "Only functionality already committed to trunk should be submitted to
> a sustaining release." and this functionality does not yet have a
> patch for trunk yet (let alone committed).
>
> Thanks,
> Eli
>
> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> > Hi all,
> > I propose to make a 0.20.205.1 candidate soon, with the following sets of
> > patches:
> >
> >   - deficiencies in HBase support, pointed out by the HBase team and
> others
> >   - deficiencies in webhdfs support on secure clusters
> >   - a couple last-minute fixes submitted to branch-0.20-security-205 that
> >   were too late to be included in 205.0
> >
> > If you would like other patches included, and you feel it is appropriate
> to
> > have them in 205.1 rather than waiting for 206.0, please declare them by
> > setting the "Target Versions" field in their Jiras, and they will receive
> > due consideration, assuming that the proposed patch is actually
> > contributed, tested, reviewed, approved, and committed
> > to branch-0.20-security-205 by the freeze date :-)
> >
> > I would like to make the rc0 candidate next Friday, so I propose to
> declare
> > 205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem for
> > anyone, please let me know.
> >
> > Thank you, and best regards,
> > --Matt (Release Manager)
> >
>

--
Subscribe to my posts at http://www.facebook.com/dhruba

Eli,

As Jitendra indicated in the jira, this was originally supposed to be part
of 0.205. Due to time crunc, we could not get this done in 0.205. This can
be turned off by a flag and only can be enabled by users who want to use
the functionality. Given that, I feel it is okay to go into 0.205.1.

I agree it would be good to have a trunk patch for this and make it part of
0.23.

Regards,
Suresh

On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]> wrote:

> Hey Matt,
>
> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
> non-trivial it seems better suited for 206 than a point release. Also,
> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
> "Only functionality already committed to trunk should be submitted to
> a sustaining release." and this functionality does not yet have a
> patch for trunk yet (let alone committed).
>
> Thanks,
> Eli
>
> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> > Hi all,
> > I propose to make a 0.20.205.1 candidate soon, with the following sets of
> > patches:
> >
> >   - deficiencies in HBase support, pointed out by the HBase team and
> others
> >   - deficiencies in webhdfs support on secure clusters
> >   - a couple last-minute fixes submitted to branch-0.20-security-205 that
> >   were too late to be included in 205.0
> >
> > If you would like other patches included, and you feel it is appropriate
> to
> > have them in 205.1 rather than waiting for 206.0, please declare them by
> > setting the "Target Versions" field in their Jiras, and they will receive
> > due consideration, assuming that the proposed patch is actually
> > contributed, tested, reviewed, approved, and committed
> > to branch-0.20-security-205 by the freeze date :-)
> >
> > I would like to make the rc0 candidate next Friday, so I propose to
> declare
> > 205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem for
> > anyone, please let me know.
> >
> > Thank you, and best regards,
> > --Matt (Release Manager)
> >
>

On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
> Eli,
>
> As Jitendra indicated in the jira, this was originally supposed to be part
> of 0.205. Due to time crunc, we could not get this done in 0.205. This can
> be turned off by a flag and only can be enabled by users who want to use
> the functionality. Given that, I feel it is okay to go into 0.205.1.

I had spoken with Sanjay offline about this patch a week or so before
205 was frozen. At that point we had left the discussion that we would
try to get it in for 205 with the understanding that a trunk patch
would be posted within a day or two of the 205 patch. Given that it's
now been a month or two since then and we still haven't seen any trunk
work here, I don't understand why we're now rushing and breaking our
own release policies.

>
> I agree it would be good to have a trunk patch for this and make it part of
> 0.23.

Yes, let's make it part of 23 and *then* backport the optimization if need be.

-Todd

> On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>
>> Hey Matt,
>>
>> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
>> non-trivial it seems better suited for 206 than a point release. Also,
>> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
>> "Only functionality already committed to trunk should be submitted to
>> a sustaining release." and this functionality does not yet have a
>> patch for trunk yet (let alone committed).
>>
>> Thanks,
>> Eli
>>
>> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
>> > Hi all,
>> > I propose to make a 0.20.205.1 candidate soon, with the following sets of
>> > patches:
>> >
>> >   - deficiencies in HBase support, pointed out by the HBase team and
>> others
>> >   - deficiencies in webhdfs support on secure clusters
>> >   - a couple last-minute fixes submitted to branch-0.20-security-205 that
>> >   were too late to be included in 205.0
>> >
>> > If you would like other patches included, and you feel it is appropriate
>> to
>> > have them in 205.1 rather than waiting for 206.0, please declare them by
>> > setting the "Target Versions" field in their Jiras, and they will receive
>> > due consideration, assuming that the proposed patch is actually
>> > contributed, tested, reviewed, approved, and committed
>> > to branch-0.20-security-205 by the freeze date :-)
>> >
>> > I would like to make the rc0 candidate next Friday, so I propose to
>> declare
>> > 205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem for
>> > anyone, please let me know.
>> >
>> > Thank you, and best regards,
>> > --Matt (Release Manager)
>> >
>>
>

--
Todd Lipcon
Software Engineer, Cloudera

Hey guys,

HDFS-2246 is not a fix, it's a non-trivial performance optimization.
The roadmap page is pretty clear..  "Point releases are made to fix
critical bugs. They do not introduce new features or make other
improvements other than fixing bugs".

I'm not opposed to the change, I'm just pointing out that we agreed to
develop trunk first, and we agreed to follow the release policies for
the sustaining branch. I don't see why we can't honor those
agreements, ie why not post a patch for trunk first and then backport
it to 206? Reasonable?

Thanks,
Eli

On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
> Eli,
>
> As Jitendra indicated in the jira, this was originally supposed to be part
> of 0.205. Due to time crunc, we could not get this done in 0.205. This can
> be turned off by a flag and only can be enabled by users who want to use
> the functionality. Given that, I feel it is okay to go into 0.205.1.
>
> I agree it would be good to have a trunk patch for this and make it part of
> 0.23.
>
> Regards,
> Suresh
>
> On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>
>> Hey Matt,
>>
>> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
>> non-trivial it seems better suited for 206 than a point release. Also,
>> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
>> "Only functionality already committed to trunk should be submitted to
>> a sustaining release." and this functionality does not yet have a
>> patch for trunk yet (let alone committed).
>>
>> Thanks,
>> Eli
>>
>> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
>> > Hi all,
>> > I propose to make a 0.20.205.1 candidate soon, with the following sets of
>> > patches:
>> >
>> >   - deficiencies in HBase support, pointed out by the HBase team and
>> others
>> >   - deficiencies in webhdfs support on secure clusters
>> >   - a couple last-minute fixes submitted to branch-0.20-security-205 that
>> >   were too late to be included in 205.0
>> >
>> > If you would like other patches included, and you feel it is appropriate
>> to
>> > have them in 205.1 rather than waiting for 206.0, please declare them by
>> > setting the "Target Versions" field in their Jiras, and they will receive
>> > due consideration, assuming that the proposed patch is actually
>> > contributed, tested, reviewed, approved, and committed
>> > to branch-0.20-security-205 by the freeze date :-)
>> >
>> > I would like to make the rc0 candidate next Friday, so I propose to
>> declare
>> > 205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem for
>> > anyone, please let me know.
>> >
>> > Thank you, and best regards,
>> > --Matt (Release Manager)
>> >
>>
>

> I had spoken with Sanjay offline about this patch a week or so before
> 205 was frozen. At that point we had left the discussion that we would
> try to get it in for 205 with the understanding that a trunk patch
> would be posted within a day or two of the 205 patch. Given that it's
> now been a month or two since then and we still haven't seen any trunk
> work here, I don't understand why we're now rushing and breaking our
> own release policies.
>

The patch was ready a month or two ago. Matt sent an email about building
rc0 tomorrow (a week ago). Hence the jira has become active for almost a
week or so. Not just this jira, there are others that have been committed,
since Matt posted the email. As I said, in my email earlier, the trunk
patch will be posted as well.

>
>

Hi Eli,
The reason I am looking at HDFS-2246 for 205 is that I and a number of
Hadoop and HBase community members really want 205 to have good support for
HBase.  This performance improvement turns out to be pretty important in
order to actually have good support for HBase.  In that sense, I'm inclined
to consider it a critical fix.

However, I think you have a very valid point about trunk support.  Suresh,
can we have a concurrent submission to trunk?

Also, I believe in the HDFS-2246 Jira, Todd requested extra time to review,
due to commitments at Hadoop World.  Todd, would Monday be sufficient extra
time, so as not to slow down the anticipated release schedule too much?

Thanks, and regards,
--Matt
On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]> wrote:

> Hey guys,
>
> HDFS-2246 is not a fix, it's a non-trivial performance optimization.
> The roadmap page is pretty clear..  "Point releases are made to fix
> critical bugs. They do not introduce new features or make other
> improvements other than fixing bugs".
>
> I'm not opposed to the change, I'm just pointing out that we agreed to
> develop trunk first, and we agreed to follow the release policies for
> the sustaining branch. I don't see why we can't honor those
> agreements, ie why not post a patch for trunk first and then backport
> it to 206? Reasonable?
>
> Thanks,
> Eli
>
> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <[EMAIL PROTECTED]>
> wrote:
> > Eli,
> >
> > As Jitendra indicated in the jira, this was originally supposed to be
> part
> > of 0.205. Due to time crunc, we could not get this done in 0.205. This
> can
> > be turned off by a flag and only can be enabled by users who want to use
> > the functionality. Given that, I feel it is okay to go into 0.205.1.
> >
> > I agree it would be good to have a trunk patch for this and make it part
> of
> > 0.23.
> >
> > Regards,
> > Suresh
> >
> > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
> >
> >> Hey Matt,
> >>
> >> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
> >> non-trivial it seems better suited for 206 than a point release. Also,
> >> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
> >> "Only functionality already committed to trunk should be submitted to
> >> a sustaining release." and this functionality does not yet have a
> >> patch for trunk yet (let alone committed).
> >>
> >> Thanks,
> >> Eli
> >>
> >> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> >> > Hi all,
> >> > I propose to make a 0.20.205.1 candidate soon, with the following
> sets of
> >> > patches:
> >> >
> >> >   - deficiencies in HBase support, pointed out by the HBase team and
> >> others
> >> >   - deficiencies in webhdfs support on secure clusters
> >> >   - a couple last-minute fixes submitted to branch-0.20-security-205
> that
> >> >   were too late to be included in 205.0
> >> >
> >> > If you would like other patches included, and you feel it is
> appropriate
> >> to
> >> > have them in 205.1 rather than waiting for 206.0, please declare them
> by
> >> > setting the "Target Versions" field in their Jiras, and they will
> receive
> >> > due consideration, assuming that the proposed patch is actually
> >> > contributed, tested, reviewed, approved, and committed
> >> > to branch-0.20-security-205 by the freeze date :-)
> >> >
> >> > I would like to make the rc0 candidate next Friday, so I propose to
> >> declare
> >> > 205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem
> for
> >> > anyone, please let me know.
> >> >
> >> > Thank you, and best regards,
> >> > --Matt (Release Manager)
> >> >
> >>
> >
>

Hey Matt,

My understanding of introducing the new 4 part version scheme was to
provide stability. Eg if someone is running 0.20.205.0 wants to be
able to get just critical bug fixes (maybe they don't even use HBase)
to what they're running then they can use 205.1, .2 etc. If someone
running 205 wants something that's more than a critical bug fix (eg a
performance optimization like HDFS-2246) then they upgrade to the next
minor version (206), which still supports a path for people running
205. This allows us to serve both users (those who want just critical
bug fixes to what they're running and those who want stuff like perf
improvements for HBase), while the plan of putting everything in the
point release serves one type of user at the expense of the other.
What's the disadvantage of putting this in 206? That can be released
soon as well right? According to the wiki it was supposed to branch
last month.

Thanks,
Eli

On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]> wrote:
> Hi Eli,
> The reason I am looking at HDFS-2246 for 205 is that I and a number of
> Hadoop and HBase community members really want 205 to have good support for
> HBase.  This performance improvement turns out to be pretty important in
> order to actually have good support for HBase.  In that sense, I'm inclined
> to consider it a critical fix.
>
> However, I think you have a very valid point about trunk support.  Suresh,
> can we have a concurrent submission to trunk?
>
> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to review,
> due to commitments at Hadoop World.  Todd, would Monday be sufficient extra
> time, so as not to slow down the anticipated release schedule too much?
>
> Thanks, and regards,
> --Matt
>
>
> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>
>> Hey guys,
>>
>> HDFS-2246 is not a fix, it's a non-trivial performance optimization.
>> The roadmap page is pretty clear..  "Point releases are made to fix
>> critical bugs. They do not introduce new features or make other
>> improvements other than fixing bugs".
>>
>> I'm not opposed to the change, I'm just pointing out that we agreed to
>> develop trunk first, and we agreed to follow the release policies for
>> the sustaining branch. I don't see why we can't honor those
>> agreements, ie why not post a patch for trunk first and then backport
>> it to 206? Reasonable?
>>
>> Thanks,
>> Eli
>>
>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <[EMAIL PROTECTED]>
>> wrote:
>> > Eli,
>> >
>> > As Jitendra indicated in the jira, this was originally supposed to be
>> part
>> > of 0.205. Due to time crunc, we could not get this done in 0.205. This
>> can
>> > be turned off by a flag and only can be enabled by users who want to use
>> > the functionality. Given that, I feel it is okay to go into 0.205.1.
>> >
>> > I agree it would be good to have a trunk patch for this and make it part
>> of
>> > 0.23.
>> >
>> > Regards,
>> > Suresh
>> >
>> > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>> >
>> >> Hey Matt,
>> >>
>> >> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
>> >> non-trivial it seems better suited for 206 than a point release. Also,
>> >> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
>> >> "Only functionality already committed to trunk should be submitted to
>> >> a sustaining release." and this functionality does not yet have a
>> >> patch for trunk yet (let alone committed).
>> >>
>> >> Thanks,
>> >> Eli
>> >>
>> >> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
>> >> > Hi all,
>> >> > I propose to make a 0.20.205.1 candidate soon, with the following
>> sets of
>> >> > patches:
>> >> >
>> >> >   - deficiencies in HBase support, pointed out by the HBase team and
>> >> others
>> >> >   - deficiencies in webhdfs support on secure clusters
>> >> >   - a couple last-minute fixes submitted to branch-0.20-security-205
>> that
>> >> >   were too late to be included in 205.0

As a heads up, this will require some tweaks in downstream projects.
For example:

  https://issues.apache.org/jira/browse/HIVE-2570

Currently I cannot use Hive with Hadoop 0.20.20#.0 because of this.

Thanks.

Alejandro

On Fri, Nov 11, 2011 at 10:34 AM, Eli Collins <[EMAIL PROTECTED]> wrote:
> Hey Matt,
>
> My understanding of introducing the new 4 part version scheme was to
> provide stability. Eg if someone is running 0.20.205.0 wants to be
> able to get just critical bug fixes (maybe they don't even use HBase)
> to what they're running then they can use 205.1, .2 etc. If someone
> running 205 wants something that's more than a critical bug fix (eg a
> performance optimization like HDFS-2246) then they upgrade to the next
> minor version (206), which still supports a path for people running
> 205. This allows us to serve both users (those who want just critical
> bug fixes to what they're running and those who want stuff like perf
> improvements for HBase), while the plan of putting everything in the
> point release serves one type of user at the expense of the other.
> What's the disadvantage of putting this in 206? That can be released
> soon as well right? According to the wiki it was supposed to branch
> last month.
>
> Thanks,
> Eli
>
> On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]> wrote:
>> Hi Eli,
>> The reason I am looking at HDFS-2246 for 205 is that I and a number of
>> Hadoop and HBase community members really want 205 to have good support for
>> HBase.  This performance improvement turns out to be pretty important in
>> order to actually have good support for HBase.  In that sense, I'm inclined
>> to consider it a critical fix.
>>
>> However, I think you have a very valid point about trunk support.  Suresh,
>> can we have a concurrent submission to trunk?
>>
>> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to review,
>> due to commitments at Hadoop World.  Todd, would Monday be sufficient extra
>> time, so as not to slow down the anticipated release schedule too much?
>>
>> Thanks, and regards,
>> --Matt
>>
>>
>> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>>
>>> Hey guys,
>>>
>>> HDFS-2246 is not a fix, it's a non-trivial performance optimization.
>>> The roadmap page is pretty clear..  "Point releases are made to fix
>>> critical bugs. They do not introduce new features or make other
>>> improvements other than fixing bugs".
>>>
>>> I'm not opposed to the change, I'm just pointing out that we agreed to
>>> develop trunk first, and we agreed to follow the release policies for
>>> the sustaining branch. I don't see why we can't honor those
>>> agreements, ie why not post a patch for trunk first and then backport
>>> it to 206? Reasonable?
>>>
>>> Thanks,
>>> Eli
>>>
>>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <[EMAIL PROTECTED]>
>>> wrote:
>>> > Eli,
>>> >
>>> > As Jitendra indicated in the jira, this was originally supposed to be
>>> part
>>> > of 0.205. Due to time crunc, we could not get this done in 0.205. This
>>> can
>>> > be turned off by a flag and only can be enabled by users who want to use
>>> > the functionality. Given that, I feel it is okay to go into 0.205.1.
>>> >
>>> > I agree it would be good to have a trunk patch for this and make it part
>>> of
>>> > 0.23.
>>> >
>>> > Regards,
>>> > Suresh
>>> >
>>> > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>>> >
>>> >> Hey Matt,
>>> >>
>>> >> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
>>> >> non-trivial it seems better suited for 206 than a point release. Also,
>>> >> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
>>> >> "Only functionality already committed to trunk should be submitted to
>>> >> a sustaining release." and this functionality does not yet have a
>>> >> patch for trunk yet (let alone committed).
>>> >>
>>> >> Thanks,
>>> >> Eli
>>> >>
>>> >> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:

Hi Matt,

I propose the following for consideration for 0.20.205.1:

https://issues.apache.org/jira/browse/HADOOP-7816 Allow Hadoop warning suppression to be specified in config
+ Improves workaround for a common usability complaint

https://issues.apache.org/jira/browse/MAPREDUCE-2980- Fetch failures and other related issues in Jetty 6.1.26.
+ The issues continues to be a source of significant operational pain

https://issues.apache.org/jira/browse/HADOOP-7810 Move Hadoop archive to core from tools
+ Improves integration with pig and hcat that wish to use hadoop archives without having to specifically callout the hadoop tools jar

https://issues.apache.org/jira/browse/mapreduce-3118 Backport Gridmix and Rumen features from trunk to Hadoop 20-Security
+ Originally slated  for 205.0 but missed original code freeze
+ Improves testability

Thanks
Nathan

On 11/9/11 8:36 PM, "Matt Foley" <[EMAIL PROTECTED]> wrote:

Hi all,
*So far, the following patches have been committed and accepted for
inclusion in 0.20.205.1:*

   *Key* *Assignee* *Reporter* *Summary*
HADOOP-5124<https://issues.apache.org/jira/browse/HADOOP-5124> Hairong
Kuang Hairong Kuang A few optimizations to FsNamesystem#RecentInvalidateSets
HADOOP-6840 <https://issues.apache.org/jira/browse/HADOOP-6840> Jitendra
Nath Pandey Nicolas Spiegelberg Support non-recursive create() in
FileSystem & SequenceFile.Writer
HADOOP-6886<https://issues.apache.org/jira/browse/HADOOP-6886>
Unassigned Nicolas Spiegelberg LocalFileSystem Needs createNonRecursive API
HADOOP-7664 <https://issues.apache.org/jira/browse/HADOOP-7664> Ravi
Prakash Ravi
Prakash o.a.h.conf.Configuration complains of overriding final parameter
even if the value with which its attempting to override is the same.
HADOOP-7711 <https://issues.apache.org/jira/browse/HADOOP-7711> Arpit
Gupta Arpit
Gupta hadoop-env.sh generated from templates has duplicate info
HADOOP-7715<https://issues.apache.org/jira/browse/HADOOP-7715> Eric
Yang Arpit Gupta see log4j Error when running mr jobs and certain dfs calls
HADOOP-7728 <https://issues.apache.org/jira/browse/HADOOP-7728> Ramya
Sunil Ramya
Sunil hadoop-setup-conf.sh should be modified to enable task memory manager
HADOOP-7740 <https://issues.apache.org/jira/browse/HADOOP-7740> Arpit
Gupta Arpit
Gupta security audit logger is not on by default, fix the log4j properties
to enable the logger
HADOOP-7765<https://issues.apache.org/jira/browse/HADOOP-7765> Eric
Yang Eric Yang Debian package contain both system and tar ball layout
HADOOP-7784 <https://issues.apache.org/jira/browse/HADOOP-7784> Eric Yang Arpit
Gupta secure datanodes fail to come up stating jsvc not found
HDFS-611<https://issues.apache.org/jira/browse/HDFS-611> Zheng
Shao dhruba borthakur Heartbeats times from Datanodes increase when there
are plenty of blocks to delete
HDFS-617<https://issues.apache.org/jira/browse/HDFS-617> Kan
Zhang Kan Zhang Support for non-recursive create() in HDFS
HDFS-1257<https://issues.apache.org/jira/browse/HDFS-1257> Eric
Payne Ramkumar Vadali Race condition on FSNamesystem#recentInvalidateSets
introduced by HADOOP-5124
HDFS-1943<https://issues.apache.org/jira/browse/HDFS-1943> Matt
Foley Wei Yongjun fail to start datanode while start-dfs.sh is executed by
root user  HDFS-2065 <https://issues.apache.org/jira/browse/HDFS-2065> Uma
Maheswara Rao G Bharath Mundlapudi Fix NPE in DFSClient.getFileChecksum
HDFS-2316 <https://issues.apache.org/jira/browse/HDFS-2316> Tsz Wo
(Nicholas), SZE Tsz Wo (Nicholas), SZE webhdfs: a complete FileSystem
implementation for accessing HDFS over HTTP [[and sub-tasks]]
HDFS-2450<https://issues.apache.org/jira/browse/HDFS-2450> Daryn
Sharp Rajit Saha Only complete hostname is supported to access data via
hdfs://  MAPREDUCE-3374<https://issues.apache.org/jira/browse/MAPREDUCE-3374>
Unassigned Roman Shaposhnik src/c++/task-controller/configure is not set
executable in the tarball and that prevents task-controller from rebuilding

*The following Jiras are targetted for 0.20.205.1, but have not yet been
committed:*

   *Key* *Assignee* *Reporter* *Summary*
HADOOP-6453<https://issues.apache.org/jira/browse/HADOOP-6453> Matt
Foley Chad Metcalf Hadoop wrapper script shouldn't ignore an existing
JAVA_LIBRARY_PATH
HADOOP-7732<https://issues.apache.org/jira/browse/HADOOP-7732>
Unassigned Arpit Gupta hadoop java docs missing hdfs package
HADOOP-7751<https://issues.apache.org/jira/browse/HADOOP-7751> Eric
Yang Arpit Gupta rpm version is not being picked from the -Dversion option
in 205  HADOOP-7804 <https://issues.apache.org/jira/browse/HADOOP-7804> Arpit
Gupta Arpit Gupta enable hadoop config generator to set
dfs.block.local-path-access.user to enable short circuit read
HADOOP-7809<https://issues.apache.org/jira/browse/HADOOP-7809> Matt
Foley Joydeep Sen Sarma Backport HADOOP-5839 to 0.20-security - fixes to
ec2 scripts to allow remote job submission
HDFS-2246<https://issues.apache.org/jira/browse/HDFS-2246>
Unassigned Sanjay Radia Shortcut a local client reads to a Datanodes files
directly  HDFS-2433 <https://issues.apache.org/jira/browse/HDFS-2433>
Unassigned Robert Joseph Evans TestFileAppend4 fails intermittently
MAPREDUCE-3343 <https://issues.apache.org/jira/browse/MAPREDUCE-3343>
Unassigned Ahmed Radwan TaskTracker Out of Memory because of distributed
cache

*Please review the items you are interested in, and complete all items by
Friday for the rc0 build.*

Also, two unit tests are failing in the nightly builds.  They passed in
Build #70 (Nov 4, 2011 12:47:00 AM GMT).
Please take a look if you committed to 205 branch after that time:
    org.apache.hadoop.streaming.TestUlimit.testCommandLine<https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-0.20.205-Build/76/testReport/org.apache.hadoop.streaming/TestUlimit/testCommandLine/>

    org.apache.hadoop.mapred.TestJvmReuse.testTaskLogs<https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-0.20.205-Build/76/testReport/org.apache.hadoop.mapred/TestJvmReuse/testTaskLogs/>

On Fri, Nov 11, 2011 at 1:41 PM, Nathan Roberts <[EMAIL PROTECTED]> wrote:
> Hi Matt,
>
> I propose the following for consideration for 0.20.205.1:
>
> https://issues.apache.org/jira/browse/HADOOP-7816 Allow Hadoop warning suppression to be specified in config
> + Improves workaround for a common usability complaint
>
> https://issues.apache.org/jira/browse/MAPREDUCE-2980- Fetch failures and other related issues in Jetty 6.1.26.
> + The issues continues to be a source of significant operational pain
>

The jetty version proposed in that JIRA doesn't actually fix the
problem entirely. We've seen at least one more case of a spinning TT
in test clusters with the "6.1.26.1" version mentioned there. The
workaround we plan to use in CDH is MAPREDUCE-3184 (TT will monitor
itself for this condition and commit suicide when it finds it)

> https://issues.apache.org/jira/browse/HADOOP-7810 Move Hadoop archive to core from tools
> + Improves integration with pig and hcat that wish to use hadoop archives without having to specifically callout the hadoop tools jar

This seems like an incompatible change - probably a bad idea for a dot
release of a dot release of a maintenance series....

>
> https://issues.apache.org/jira/browse/mapreduce-3118 Backport Gridmix and Rumen features from trunk to Hadoop 20-Security
> + Originally slated  for 205.0 but missed original code freeze
> + Improves testability
>
> Thanks
> Nathan
>
> On 11/9/11 8:36 PM, "Matt Foley" <[EMAIL PROTECTED]> wrote:
>
> Hi all,
> *So far, the following patches have been committed and accepted for
> inclusion in 0.20.205.1:*
>
>   *Key* *Assignee* *Reporter* *Summary*
> HADOOP-5124<https://issues.apache.org/jira/browse/HADOOP-5124> Hairong
> Kuang Hairong Kuang A few optimizations to FsNamesystem#RecentInvalidateSets
> HADOOP-6840 <https://issues.apache.org/jira/browse/HADOOP-6840> Jitendra
> Nath Pandey Nicolas Spiegelberg Support non-recursive create() in
> FileSystem & SequenceFile.Writer
> HADOOP-6886<https://issues.apache.org/jira/browse/HADOOP-6886>
> Unassigned Nicolas Spiegelberg LocalFileSystem Needs createNonRecursive API
> HADOOP-7664 <https://issues.apache.org/jira/browse/HADOOP-7664> Ravi
> Prakash Ravi
> Prakash o.a.h.conf.Configuration complains of overriding final parameter
> even if the value with which its attempting to override is the same.
> HADOOP-7711 <https://issues.apache.org/jira/browse/HADOOP-7711> Arpit
> Gupta Arpit
> Gupta hadoop-env.sh generated from templates has duplicate info
> HADOOP-7715<https://issues.apache.org/jira/browse/HADOOP-7715> Eric
> Yang Arpit Gupta see log4j Error when running mr jobs and certain dfs calls
> HADOOP-7728 <https://issues.apache.org/jira/browse/HADOOP-7728> Ramya
> Sunil Ramya
> Sunil hadoop-setup-conf.sh should be modified to enable task memory manager
> HADOOP-7740 <https://issues.apache.org/jira/browse/HADOOP-7740> Arpit
> Gupta Arpit
> Gupta security audit logger is not on by default, fix the log4j properties
> to enable the logger
> HADOOP-7765<https://issues.apache.org/jira/browse/HADOOP-7765> Eric
> Yang Eric Yang Debian package contain both system and tar ball layout
> HADOOP-7784 <https://issues.apache.org/jira/browse/HADOOP-7784> Eric Yang Arpit
> Gupta secure datanodes fail to come up stating jsvc not found
> HDFS-611<https://issues.apache.org/jira/browse/HDFS-611> Zheng
> Shao dhruba borthakur Heartbeats times from Datanodes increase when there
> are plenty of blocks to delete
> HDFS-617<https://issues.apache.org/jira/browse/HDFS-617> Kan
> Zhang Kan Zhang Support for non-recursive create() in HDFS
> HDFS-1257<https://issues.apache.org/jira/browse/HDFS-1257> Eric
> Payne Ramkumar Vadali Race condition on FSNamesystem#recentInvalidateSets
> introduced by HADOOP-5124
> HDFS-1943<https://issues.apache.org/jira/browse/HDFS-1943> Matt
> Foley Wei Yongjun fail to start datanode while start-dfs.sh is executed by
> root user  HDFS-2065 <https://issues.apache.org/jira/browse/HDFS-2065> Uma
> Maheswara Rao G Bharath Mundlapudi Fix NPE in DFSClient.getFileChecksum

Todd Lipcon
Software Engineer, Cloudera

On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]> wrote:

>
> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to review,
> due to commitments at Hadoop World.  Todd, would Monday be sufficient extra
> time, so as not to slow down the anticipated release schedule too much?
>

Yes, I will probably have time to review it by Monday. But the
review-time concern is separate from the concern about which version
this should go into.

Calling this a "critical fix" for HBase is a bit strange as 99.9% of
the HBase installs out there do not use it. Trend Micro and Facebook
are the only ones I'm aware of that do. And the patch as it stands
today has a very suspect security model...

-Todd
>
> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>
>> Hey guys,
>>
>> HDFS-2246 is not a fix, it's a non-trivial performance optimization.
>> The roadmap page is pretty clear..  "Point releases are made to fix
>> critical bugs. They do not introduce new features or make other
>> improvements other than fixing bugs".
>>
>> I'm not opposed to the change, I'm just pointing out that we agreed to
>> develop trunk first, and we agreed to follow the release policies for
>> the sustaining branch. I don't see why we can't honor those
>> agreements, ie why not post a patch for trunk first and then backport
>> it to 206? Reasonable?
>>
>> Thanks,
>> Eli
>>
>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <[EMAIL PROTECTED]>
>> wrote:
>> > Eli,
>> >
>> > As Jitendra indicated in the jira, this was originally supposed to be
>> part
>> > of 0.205. Due to time crunc, we could not get this done in 0.205. This
>> can
>> > be turned off by a flag and only can be enabled by users who want to use
>> > the functionality. Given that, I feel it is okay to go into 0.205.1.
>> >
>> > I agree it would be good to have a trunk patch for this and make it part
>> of
>> > 0.23.
>> >
>> > Regards,
>> > Suresh
>> >
>> > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>> >
>> >> Hey Matt,
>> >>
>> >> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
>> >> non-trivial it seems better suited for 206 than a point release. Also,
>> >> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
>> >> "Only functionality already committed to trunk should be submitted to
>> >> a sustaining release." and this functionality does not yet have a
>> >> patch for trunk yet (let alone committed).
>> >>
>> >> Thanks,
>> >> Eli
>> >>
>> >> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
>> >> > Hi all,
>> >> > I propose to make a 0.20.205.1 candidate soon, with the following
>> sets of
>> >> > patches:
>> >> >
>> >> >   - deficiencies in HBase support, pointed out by the HBase team and
>> >> others
>> >> >   - deficiencies in webhdfs support on secure clusters
>> >> >   - a couple last-minute fixes submitted to branch-0.20-security-205
>> that
>> >> >   were too late to be included in 205.0
>> >> >
>> >> > If you would like other patches included, and you feel it is
>> appropriate
>> >> to
>> >> > have them in 205.1 rather than waiting for 206.0, please declare them
>> by
>> >> > setting the "Target Versions" field in their Jiras, and they will
>> receive
>> >> > due consideration, assuming that the proposed patch is actually
>> >> > contributed, tested, reviewed, approved, and committed
>> >> > to branch-0.20-security-205 by the freeze date :-)
>> >> >
>> >> > I would like to make the rc0 candidate next Friday, so I propose to
>> >> declare
>> >> > 205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem
>> for
>> >> > anyone, please let me know.
>> >> >
>> >> > Thank you, and best regards,
>> >> > --Matt (Release Manager)
>> >> >
>> >>
>> >
>>
>

--
Todd Lipcon
Software Engineer, Cloudera

> From: Todd Lipcon <[EMAIL PROTECTED]>

> Calling this a "critical fix" for HBase is a bit strange as 99.9% of
> the HBase installs out there do not use it. Trend Micro and Facebook
> are the only ones I'm aware of that do.

It would be more accurate to say we are running it in one production installation, but

  - have questions as to the performance benefits we will actually see

  - won't be able to use it in a deployment that requires stronger assurance 

> And the patch as it stands today has a very suspect security model...
Agreed.

In my opinion, this is a useful option to provide, but off by default, and isn't a critical fix. Nothing is broken. Call it "a performance optimization option".

Best regards,
   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Todd Lipcon <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Friday, November 11, 2011 5:29 PM
> Subject: Re: [ANNOUNCE] Intend to build a 0.20.205.1 candidate next Friday 11 Nov.
>
> On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]>
> wrote:
>
>>
>>  Also, I believe in the HDFS-2246 Jira, Todd requested extra time to review,
>>  due to commitments at Hadoop World.  Todd, would Monday be sufficient extra
>>  time, so as not to slow down the anticipated release schedule too much?
>>
>
> Yes, I will probably have time to review it by Monday. But the
> review-time concern is separate from the concern about which version
> this should go into.
>
> Calling this a "critical fix" for HBase is a bit strange as 99.9% of
> the HBase installs out there do not use it. Trend Micro and Facebook
> are the only ones I'm aware of that do. And the patch as it stands
> today has a very suspect security model...
>
> -Todd
>
>
>>
>>  On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]>
> wrote:
>>
>>>  Hey guys,
>>>
>>>  HDFS-2246 is not a fix, it's a non-trivial performance
> optimization.
>>>  The roadmap page is pretty clear..  "Point releases are made to
> fix
>>>  critical bugs. They do not introduce new features or make other
>>>  improvements other than fixing bugs".
>>>
>>>  I'm not opposed to the change, I'm just pointing out that we
> agreed to
>>>  develop trunk first, and we agreed to follow the release policies for
>>>  the sustaining branch. I don't see why we can't honor those
>>>  agreements, ie why not post a patch for trunk first and then backport
>>>  it to 206? Reasonable?
>>>
>>>  Thanks,
>>>  Eli
>>>
>>>  On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas
> <[EMAIL PROTECTED]>
>>>  wrote:
>>>  > Eli,
>>>  >
>>>  > As Jitendra indicated in the jira, this was originally supposed to
> be
>>>  part
>>>  > of 0.205. Due to time crunc, we could not get this done in 0.205.
> This
>>>  can
>>>  > be turned off by a flag and only can be enabled by users who want
> to use
>>>  > the functionality. Given that, I feel it is okay to go into
> 0.205.1.
>>>  >
>>>  > I agree it would be good to have a trunk patch for this and make
> it part
>>>  of
>>>  > 0.23.
>>>  >
>>>  > Regards,
>>>  > Suresh
>>>  >
>>>  > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins
> <[EMAIL PROTECTED]> wrote:
>>>  >
>>>  >> Hey Matt,
>>>  >>
>>>  >> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a
> bug and is
>>>  >> non-trivial it seems better suited for 206 than a point
> release. Also,
>>>  >> per the sustaining roadmap -
> http://wiki.apache.org/hadoop/Roadmap -
>>>  >> "Only functionality already committed to trunk should be
> submitted to
>>>  >> a sustaining release." and this functionality does not
> yet have a
>>>  >> patch for trunk yet (let alone committed).
>>>  >>
>>>  >> Thanks,
>>>  >> Eli
>>>  >>
>>>  >> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley
> <[EMAIL PROTECTED]> wrote:
>>>  >> > Hi all,
>>>  >> > I propose to make a 0.20.205.1 candidate soon, with the
> following
>>>  sets of
>>>  >> > patches:
>>>  

On Fri, Nov 11, 2011 at 2:29 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]> wrote:
>
>>
>> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to review,
>> due to commitments at Hadoop World.  Todd, would Monday be sufficient extra
>> time, so as not to slow down the anticipated release schedule too much?
>>
>
> Yes, I will probably have time to review it by Monday. But the
> review-time concern is separate from the concern about which version
> this should go into.
>

Reviewing this now... though I still think it shoudl target 0.20.206,
not 0.20.205.1.

-Todd

>
>>
>> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>>
>>> Hey guys,
>>>
>>> HDFS-2246 is not a fix, it's a non-trivial performance optimization.
>>> The roadmap page is pretty clear..  "Point releases are made to fix
>>> critical bugs. They do not introduce new features or make other
>>> improvements other than fixing bugs".
>>>
>>> I'm not opposed to the change, I'm just pointing out that we agreed to
>>> develop trunk first, and we agreed to follow the release policies for
>>> the sustaining branch. I don't see why we can't honor those
>>> agreements, ie why not post a patch for trunk first and then backport
>>> it to 206? Reasonable?
>>>
>>> Thanks,
>>> Eli
>>>
>>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <[EMAIL PROTECTED]>
>>> wrote:
>>> > Eli,
>>> >
>>> > As Jitendra indicated in the jira, this was originally supposed to be
>>> part
>>> > of 0.205. Due to time crunc, we could not get this done in 0.205. This
>>> can
>>> > be turned off by a flag and only can be enabled by users who want to use
>>> > the functionality. Given that, I feel it is okay to go into 0.205.1.
>>> >
>>> > I agree it would be good to have a trunk patch for this and make it part
>>> of
>>> > 0.23.
>>> >
>>> > Regards,
>>> > Suresh
>>> >
>>> > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>>> >
>>> >> Hey Matt,
>>> >>
>>> >> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and is
>>> >> non-trivial it seems better suited for 206 than a point release. Also,
>>> >> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap -
>>> >> "Only functionality already committed to trunk should be submitted to
>>> >> a sustaining release." and this functionality does not yet have a
>>> >> patch for trunk yet (let alone committed).
>>> >>
>>> >> Thanks,
>>> >> Eli
>>> >>
>>> >> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
>>> >> > Hi all,
>>> >> > I propose to make a 0.20.205.1 candidate soon, with the following
>>> sets of
>>> >> > patches:
>>> >> >
>>> >> >   - deficiencies in HBase support, pointed out by the HBase team and
>>> >> others
>>> >> >   - deficiencies in webhdfs support on secure clusters
>>> >> >   - a couple last-minute fixes submitted to branch-0.20-security-205
>>> that
>>> >> >   were too late to be included in 205.0
>>> >> >
>>> >> > If you would like other patches included, and you feel it is
>>> appropriate
>>> >> to
>>> >> > have them in 205.1 rather than waiting for 206.0, please declare them
>>> by
>>> >> > setting the "Target Versions" field in their Jiras, and they will
>>> receive
>>> >> > due consideration, assuming that the proposed patch is actually
>>> >> > contributed, tested, reviewed, approved, and committed
>>> >> > to branch-0.20-security-205 by the freeze date :-)
>>> >> >
>>> >> > I would like to make the rc0 candidate next Friday, so I propose to
>>> >> declare
>>> >> > 205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a problem
>>> for
>>> >> > anyone, please let me know.
>>> >> >
>>> >> > Thank you, and best regards,
>>> >> > --Matt (Release Manager)
>>> >> >
>>> >>
>>> >
>>>
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--
Todd Lipcon
Software Engineer, Cloudera

The trunk, 206 patches for HDFS-2246 have been committed. I think it makes
sense to commit it to 205.1 as well for following reasons (most of it has
already been mentioned)
a) We intended this patch for 205, but couldn't finish in time. Now that
205.1 branch is still not cut, we could get this in.
b) This is not a very risky change. Most of it is new code and will be
disabled by default the feature will be disabled.
c) The performance benefits are very good, as reported by Todd on the jira.
Hbase installations will significantly benefit from it.

On Mon, Nov 14, 2011 at 1:44 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Fri, Nov 11, 2011 at 2:29 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> > On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]>
> wrote:
> >
> >>
> >> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to
> review,
> >> due to commitments at Hadoop World.  Todd, would Monday be sufficient
> extra
> >> time, so as not to slow down the anticipated release schedule too much?
> >>
> >
> > Yes, I will probably have time to review it by Monday. But the
> > review-time concern is separate from the concern about which version
> > this should go into.
> >
>
> Reviewing this now... though I still think it shoudl target 0.20.206,
> not 0.20.205.1.
>
> -Todd
>
> >
> >>
> >> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
> >>
> >>> Hey guys,
> >>>
> >>> HDFS-2246 is not a fix, it's a non-trivial performance optimization.
> >>> The roadmap page is pretty clear..  "Point releases are made to fix
> >>> critical bugs. They do not introduce new features or make other
> >>> improvements other than fixing bugs".
> >>>
> >>> I'm not opposed to the change, I'm just pointing out that we agreed to
> >>> develop trunk first, and we agreed to follow the release policies for
> >>> the sustaining branch. I don't see why we can't honor those
> >>> agreements, ie why not post a patch for trunk first and then backport
> >>> it to 206? Reasonable?
> >>>
> >>> Thanks,
> >>> Eli
> >>>
> >>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <
> [EMAIL PROTECTED]>
> >>> wrote:
> >>> > Eli,
> >>> >
> >>> > As Jitendra indicated in the jira, this was originally supposed to be
> >>> part
> >>> > of 0.205. Due to time crunc, we could not get this done in 0.205.
> This
> >>> can
> >>> > be turned off by a flag and only can be enabled by users who want to
> use
> >>> > the functionality. Given that, I feel it is okay to go into 0.205.1.
> >>> >
> >>> > I agree it would be good to have a trunk patch for this and make it
> part
> >>> of
> >>> > 0.23.
> >>> >
> >>> > Regards,
> >>> > Suresh
> >>> >
> >>> > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]>
> wrote:
> >>> >
> >>> >> Hey Matt,
> >>> >>
> >>> >> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and
> is
> >>> >> non-trivial it seems better suited for 206 than a point release.
> Also,
> >>> >> per the sustaining roadmap - http://wiki.apache.org/hadoop/Roadmap-
> >>> >> "Only functionality already committed to trunk should be submitted
> to
> >>> >> a sustaining release." and this functionality does not yet have a
> >>> >> patch for trunk yet (let alone committed).
> >>> >>
> >>> >> Thanks,
> >>> >> Eli
> >>> >>
> >>> >> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]>
> wrote:
> >>> >> > Hi all,
> >>> >> > I propose to make a 0.20.205.1 candidate soon, with the following
> >>> sets of
> >>> >> > patches:
> >>> >> >
> >>> >> >   - deficiencies in HBase support, pointed out by the HBase team
> and
> >>> >> others
> >>> >> >   - deficiencies in webhdfs support on secure clusters
> >>> >> >   - a couple last-minute fixes submitted to
> branch-0.20-security-205
> >>> that
> >>> >> >   were too late to be included in 205.0
> >>> >> >
> >>> >> > If you would like other patches included, and you feel it is
> >>> appropriate
> >>> >> to
> >>> >> > have them in 205.1 rather than waiting for 206.0, please declare

+1 for Jitendra's proposal.

Additionally, most of the core of the code that this patch is based on has
been tested and deployed in clusters at TrendMicro and Facebook.

On Wed, Nov 23, 2011 at 11:10 AM, Jitendra Pandey
<[EMAIL PROTECTED]>wrote:

> The trunk, 206 patches for HDFS-2246 have been committed. I think it makes
> sense to commit it to 205.1 as well for following reasons (most of it has
> already been mentioned)
> a) We intended this patch for 205, but couldn't finish in time. Now that
> 205.1 branch is still not cut, we could get this in.
> b) This is not a very risky change. Most of it is new code and will be
> disabled by default the feature will be disabled.
> c) The performance benefits are very good, as reported by Todd on the jira.
> Hbase installations will significantly benefit from it.
>
> On Mon, Nov 14, 2011 at 1:44 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
> > On Fri, Nov 11, 2011 at 2:29 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> > > On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >>
> > >> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to
> > review,
> > >> due to commitments at Hadoop World.  Todd, would Monday be sufficient
> > extra
> > >> time, so as not to slow down the anticipated release schedule too
> much?
> > >>
> > >
> > > Yes, I will probably have time to review it by Monday. But the
> > > review-time concern is separate from the concern about which version
> > > this should go into.
> > >
> >
> > Reviewing this now... though I still think it shoudl target 0.20.206,
> > not 0.20.205.1.
> >
> > -Todd
> >
> > >
> > >>
> > >> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]>
> wrote:
> > >>
> > >>> Hey guys,
> > >>>
> > >>> HDFS-2246 is not a fix, it's a non-trivial performance optimization.
> > >>> The roadmap page is pretty clear..  "Point releases are made to fix
> > >>> critical bugs. They do not introduce new features or make other
> > >>> improvements other than fixing bugs".
> > >>>
> > >>> I'm not opposed to the change, I'm just pointing out that we agreed
> to
> > >>> develop trunk first, and we agreed to follow the release policies for
> > >>> the sustaining branch. I don't see why we can't honor those
> > >>> agreements, ie why not post a patch for trunk first and then backport
> > >>> it to 206? Reasonable?
> > >>>
> > >>> Thanks,
> > >>> Eli
> > >>>
> > >>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <
> > [EMAIL PROTECTED]>
> > >>> wrote:
> > >>> > Eli,
> > >>> >
> > >>> > As Jitendra indicated in the jira, this was originally supposed to
> be
> > >>> part
> > >>> > of 0.205. Due to time crunc, we could not get this done in 0.205.
> > This
> > >>> can
> > >>> > be turned off by a flag and only can be enabled by users who want
> to
> > use
> > >>> > the functionality. Given that, I feel it is okay to go into
> 0.205.1.
> > >>> >
> > >>> > I agree it would be good to have a trunk patch for this and make it
> > part
> > >>> of
> > >>> > 0.23.
> > >>> >
> > >>> > Regards,
> > >>> > Suresh
> > >>> >
> > >>> > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]>
> > wrote:
> > >>> >
> > >>> >> Hey Matt,
> > >>> >>
> > >>> >> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug and
> > is
> > >>> >> non-trivial it seems better suited for 206 than a point release.
> > Also,
> > >>> >> per the sustaining roadmap -
> http://wiki.apache.org/hadoop/Roadmap-
> > >>> >> "Only functionality already committed to trunk should be submitted
> > to
> > >>> >> a sustaining release." and this functionality does not yet have a
> > >>> >> patch for trunk yet (let alone committed).
> > >>> >>
> > >>> >> Thanks,
> > >>> >> Eli
> > >>> >>
> > >>> >> On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]>
> > wrote:
> > >>> >> > Hi all,
> > >>> >> > I propose to make a 0.20.205.1 candidate soon, with the
> following
> > >>> sets of
> > >>> >> > patches:
> > >>> >> >
> > >>> >> >   - deficiencies in HBase support, pointed out by the HBase team

I really want this in 0.20.205.1, which will be Hadoop 1.0.0, because of
its importance for
good support of HBase.

Jitendra, please merge it to branch-0.20-security-205.

--Matt (wearing my Apache release manager hat)
On Wed, Nov 23, 2011 at 11:14 AM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> +1 for Jitendra's proposal.
>
> Additionally, most of the core of the code that this patch is based on has
> been tested and deployed in clusters at TrendMicro and Facebook.
>
> On Wed, Nov 23, 2011 at 11:10 AM, Jitendra Pandey
> <[EMAIL PROTECTED]>wrote:
>
> > The trunk, 206 patches for HDFS-2246 have been committed. I think it
> makes
> > sense to commit it to 205.1 as well for following reasons (most of it has
> > already been mentioned)
> > a) We intended this patch for 205, but couldn't finish in time. Now that
> > 205.1 branch is still not cut, we could get this in.
> > b) This is not a very risky change. Most of it is new code and will be
> > disabled by default the feature will be disabled.
> > c) The performance benefits are very good, as reported by Todd on the
> jira.
> > Hbase installations will significantly benefit from it.
> >
> > On Mon, Nov 14, 2011 at 1:44 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >
> > > On Fri, Nov 11, 2011 at 2:29 PM, Todd Lipcon <[EMAIL PROTECTED]>
> wrote:
> > > > On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > >>
> > > >> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to
> > > review,
> > > >> due to commitments at Hadoop World.  Todd, would Monday be
> sufficient
> > > extra
> > > >> time, so as not to slow down the anticipated release schedule too
> > much?
> > > >>
> > > >
> > > > Yes, I will probably have time to review it by Monday. But the
> > > > review-time concern is separate from the concern about which version
> > > > this should go into.
> > > >
> > >
> > > Reviewing this now... though I still think it shoudl target 0.20.206,
> > > not 0.20.205.1.
> > >
> > > -Todd
> > >
> > > >
> > > >>
> > > >> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]>
> > wrote:
> > > >>
> > > >>> Hey guys,
> > > >>>
> > > >>> HDFS-2246 is not a fix, it's a non-trivial performance
> optimization.
> > > >>> The roadmap page is pretty clear..  "Point releases are made to fix
> > > >>> critical bugs. They do not introduce new features or make other
> > > >>> improvements other than fixing bugs".
> > > >>>
> > > >>> I'm not opposed to the change, I'm just pointing out that we agreed
> > to
> > > >>> develop trunk first, and we agreed to follow the release policies
> for
> > > >>> the sustaining branch. I don't see why we can't honor those
> > > >>> agreements, ie why not post a patch for trunk first and then
> backport
> > > >>> it to 206? Reasonable?
> > > >>>
> > > >>> Thanks,
> > > >>> Eli
> > > >>>
> > > >>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <
> > > [EMAIL PROTECTED]>
> > > >>> wrote:
> > > >>> > Eli,
> > > >>> >
> > > >>> > As Jitendra indicated in the jira, this was originally supposed
> to
> > be
> > > >>> part
> > > >>> > of 0.205. Due to time crunc, we could not get this done in 0.205.
> > > This
> > > >>> can
> > > >>> > be turned off by a flag and only can be enabled by users who want
> > to
> > > use
> > > >>> > the functionality. Given that, I feel it is okay to go into
> > 0.205.1.
> > > >>> >
> > > >>> > I agree it would be good to have a trunk patch for this and make
> it
> > > part
> > > >>> of
> > > >>> > 0.23.
> > > >>> >
> > > >>> > Regards,
> > > >>> > Suresh
> > > >>> >
> > > >>> > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]>
> > > wrote:
> > > >>> >
> > > >>> >> Hey Matt,
> > > >>> >>
> > > >>> >> Is HDFS-2246 slated for 0.20.205.1?  Given that it's not a bug
> and
> > > is
> > > >>> >> non-trivial it seems better suited for 206 than a point release.
> > > Also,
> > > >>> >> per the sustaining roadmap -
> > http://wiki.apache.org/hadoop/Roadmap-
> > >

Hey Matt,

On the jira Jitendra referenced a 205.1 deadline. Where did you set or
communicate that deadline? The  last I saw on the lists (this thread)
was that the 205.1 code freeze was Nov 11th.

Thanks,
Eli

On Wed, Nov 23, 2011 at 2:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> I really want this in 0.20.205.1, which will be Hadoop 1.0.0, because of
> its importance for
> good support of HBase.
>
> Jitendra, please merge it to branch-0.20-security-205.
>
> --Matt (wearing my Apache release manager hat)
>
>
> On Wed, Nov 23, 2011 at 11:14 AM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:
>
>> +1 for Jitendra's proposal.
>>
>> Additionally, most of the core of the code that this patch is based on has
>> been tested and deployed in clusters at TrendMicro and Facebook.
>>
>> On Wed, Nov 23, 2011 at 11:10 AM, Jitendra Pandey
>> <[EMAIL PROTECTED]>wrote:
>>
>> > The trunk, 206 patches for HDFS-2246 have been committed. I think it
>> makes
>> > sense to commit it to 205.1 as well for following reasons (most of it has
>> > already been mentioned)
>> > a) We intended this patch for 205, but couldn't finish in time. Now that
>> > 205.1 branch is still not cut, we could get this in.
>> > b) This is not a very risky change. Most of it is new code and will be
>> > disabled by default the feature will be disabled.
>> > c) The performance benefits are very good, as reported by Todd on the
>> jira.
>> > Hbase installations will significantly benefit from it.
>> >
>> > On Mon, Nov 14, 2011 at 1:44 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>> >
>> > > On Fri, Nov 11, 2011 at 2:29 PM, Todd Lipcon <[EMAIL PROTECTED]>
>> wrote:
>> > > > On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]>
>> > > wrote:
>> > > >
>> > > >>
>> > > >> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to
>> > > review,
>> > > >> due to commitments at Hadoop World.  Todd, would Monday be
>> sufficient
>> > > extra
>> > > >> time, so as not to slow down the anticipated release schedule too
>> > much?
>> > > >>
>> > > >
>> > > > Yes, I will probably have time to review it by Monday. But the
>> > > > review-time concern is separate from the concern about which version
>> > > > this should go into.
>> > > >
>> > >
>> > > Reviewing this now... though I still think it shoudl target 0.20.206,
>> > > not 0.20.205.1.
>> > >
>> > > -Todd
>> > >
>> > > >
>> > > >>
>> > > >> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]>
>> > wrote:
>> > > >>
>> > > >>> Hey guys,
>> > > >>>
>> > > >>> HDFS-2246 is not a fix, it's a non-trivial performance
>> optimization.
>> > > >>> The roadmap page is pretty clear..  "Point releases are made to fix
>> > > >>> critical bugs. They do not introduce new features or make other
>> > > >>> improvements other than fixing bugs".
>> > > >>>
>> > > >>> I'm not opposed to the change, I'm just pointing out that we agreed
>> > to
>> > > >>> develop trunk first, and we agreed to follow the release policies
>> for
>> > > >>> the sustaining branch. I don't see why we can't honor those
>> > > >>> agreements, ie why not post a patch for trunk first and then
>> backport
>> > > >>> it to 206? Reasonable?
>> > > >>>
>> > > >>> Thanks,
>> > > >>> Eli
>> > > >>>
>> > > >>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <
>> > > [EMAIL PROTECTED]>
>> > > >>> wrote:
>> > > >>> > Eli,
>> > > >>> >
>> > > >>> > As Jitendra indicated in the jira, this was originally supposed
>> to
>> > be
>> > > >>> part
>> > > >>> > of 0.205. Due to time crunc, we could not get this done in 0.205.
>> > > This
>> > > >>> can
>> > > >>> > be turned off by a flag and only can be enabled by users who want
>> > to
>> > > use
>> > > >>> > the functionality. Given that, I feel it is okay to go into
>> > 0.205.1.
>> > > >>> >
>> > > >>> > I agree it would be good to have a trunk patch for this and make
>> it
>> > > part
>> > > >>> of
>> > > >>> > 0.23.
>> > > >>> >
>> > > >>> > Regards,
>> > > >>> > Suresh
>> > > >>> >
>> > > >>> > On Thu, Nov 10, 2011 at 4:32 PM, Eli Collins <[EMAIL PROTECTED]>

Hi Matt,

quick question: any reason we are ignoring multifilewc from hadoop examples?
   https://issues.apache.org/jira/browse/MAPREDUCE-3319

would be nice to fix it for 1.0 of Hadoop. Or at least disable.

Thanks,
Roman.

On Wed, Nov 23, 2011 at 2:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> I really want this in 0.20.205.1, which will be Hadoop 1.0.0, because of
> its importance for
> good support of HBase.
>
> Jitendra, please merge it to branch-0.20-security-205.
>
> --Matt (wearing my Apache release manager hat)
>
>
> On Wed, Nov 23, 2011 at 11:14 AM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:
>
>> +1 for Jitendra's proposal.
>>
>> Additionally, most of the core of the code that this patch is based on has
>> been tested and deployed in clusters at TrendMicro and Facebook.
>>
>> On Wed, Nov 23, 2011 at 11:10 AM, Jitendra Pandey
>> <[EMAIL PROTECTED]>wrote:
>>
>> > The trunk, 206 patches for HDFS-2246 have been committed. I think it
>> makes
>> > sense to commit it to 205.1 as well for following reasons (most of it has
>> > already been mentioned)
>> > a) We intended this patch for 205, but couldn't finish in time. Now that
>> > 205.1 branch is still not cut, we could get this in.
>> > b) This is not a very risky change. Most of it is new code and will be
>> > disabled by default the feature will be disabled.
>> > c) The performance benefits are very good, as reported by Todd on the
>> jira.
>> > Hbase installations will significantly benefit from it.
>> >
>> > On Mon, Nov 14, 2011 at 1:44 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>> >
>> > > On Fri, Nov 11, 2011 at 2:29 PM, Todd Lipcon <[EMAIL PROTECTED]>
>> wrote:
>> > > > On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]>
>> > > wrote:
>> > > >
>> > > >>
>> > > >> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to
>> > > review,
>> > > >> due to commitments at Hadoop World.  Todd, would Monday be
>> sufficient
>> > > extra
>> > > >> time, so as not to slow down the anticipated release schedule too
>> > much?
>> > > >>
>> > > >
>> > > > Yes, I will probably have time to review it by Monday. But the
>> > > > review-time concern is separate from the concern about which version
>> > > > this should go into.
>> > > >
>> > >
>> > > Reviewing this now... though I still think it shoudl target 0.20.206,
>> > > not 0.20.205.1.
>> > >
>> > > -Todd
>> > >
>> > > >
>> > > >>
>> > > >> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]>
>> > wrote:
>> > > >>
>> > > >>> Hey guys,
>> > > >>>
>> > > >>> HDFS-2246 is not a fix, it's a non-trivial performance
>> optimization.
>> > > >>> The roadmap page is pretty clear..  "Point releases are made to fix
>> > > >>> critical bugs. They do not introduce new features or make other
>> > > >>> improvements other than fixing bugs".
>> > > >>>
>> > > >>> I'm not opposed to the change, I'm just pointing out that we agreed
>> > to
>> > > >>> develop trunk first, and we agreed to follow the release policies
>> for
>> > > >>> the sustaining branch. I don't see why we can't honor those
>> > > >>> agreements, ie why not post a patch for trunk first and then
>> backport
>> > > >>> it to 206? Reasonable?
>> > > >>>
>> > > >>> Thanks,
>> > > >>> Eli
>> > > >>>
>> > > >>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <
>> > > [EMAIL PROTECTED]>
>> > > >>> wrote:
>> > > >>> > Eli,
>> > > >>> >
>> > > >>> > As Jitendra indicated in the jira, this was originally supposed
>> to
>> > be
>> > > >>> part
>> > > >>> > of 0.205. Due to time crunc, we could not get this done in 0.205.
>> > > This
>> > > >>> can
>> > > >>> > be turned off by a flag and only can be enabled by users who want
>> > to
>> > > use
>> > > >>> > the functionality. Given that, I feel it is okay to go into
>> > 0.205.1.
>> > > >>> >
>> > > >>> > I agree it would be good to have a trunk patch for this and make
>> it
>> > > part
>> > > >>> of
>> > > >>> > 0.23.
>> > > >>> >
>> > > >>> > Regards,
>> > > >>> > Suresh
>> > > >>>

Hi Eli,
I said a couple weeks ago that I intended to cut 205.1 on Nov 11 -- that's
the
subject line of this thread :-)  However, I got busy and did not make the
11/11/11
date, for which I apologize.

In the meantime, a severe blocker bug, HADOOP-7853, has been found and fixed
by folks at Yahoo, related to kerberos ticket renewal in secure
environments with Hive.
And HDFS-2246 has become available, and properly committed to trunk.
So I think it is good to incorporate both these issues.

I now plan to cut the RC for 205.1 sometime over this long weekend.
Since this thread has gotten a little off-topic, I will send a new email
announcing that.

Thanks,
--Matt

On Wed, Nov 23, 2011 at 3:05 PM, Eli Collins <[EMAIL PROTECTED]> wrote:

> Hey Matt,
>
> On the jira Jitendra referenced a 205.1 deadline. Where did you set or
> communicate that deadline? The  last I saw on the lists (this thread)
> was that the 205.1 code freeze was Nov 11th.
>
> Thanks,
> Eli
>
> On Wed, Nov 23, 2011 at 2:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> > I really want this in 0.20.205.1, which will be Hadoop 1.0.0, because of
> > its importance for
> > good support of HBase.
> >
> > Jitendra, please merge it to branch-0.20-security-205.
> >
> > --Matt (wearing my Apache release manager hat)
> >
> >
> > On Wed, Nov 23, 2011 at 11:14 AM, Suresh Srinivas <
> [EMAIL PROTECTED]>wrote:
> >
> >> +1 for Jitendra's proposal.
> >>
> >> Additionally, most of the core of the code that this patch is based on
> has
> >> been tested and deployed in clusters at TrendMicro and Facebook.
> >>
> >> On Wed, Nov 23, 2011 at 11:10 AM, Jitendra Pandey
> >> <[EMAIL PROTECTED]>wrote:
> >>
> >> > The trunk, 206 patches for HDFS-2246 have been committed. I think it
> >> makes
> >> > sense to commit it to 205.1 as well for following reasons (most of it
> has
> >> > already been mentioned)
> >> > a) We intended this patch for 205, but couldn't finish in time. Now
> that
> >> > 205.1 branch is still not cut, we could get this in.
> >> > b) This is not a very risky change. Most of it is new code and will be
> >> > disabled by default the feature will be disabled.
> >> > c) The performance benefits are very good, as reported by Todd on the
> >> jira.
> >> > Hbase installations will significantly benefit from it.
> >> >
> >> > On Mon, Nov 14, 2011 at 1:44 PM, Todd Lipcon <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > On Fri, Nov 11, 2011 at 2:29 PM, Todd Lipcon <[EMAIL PROTECTED]>
> >> wrote:
> >> > > > On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <
> [EMAIL PROTECTED]>
> >> > > wrote:
> >> > > >
> >> > > >>
> >> > > >> Also, I believe in the HDFS-2246 Jira, Todd requested extra time
> to
> >> > > review,
> >> > > >> due to commitments at Hadoop World.  Todd, would Monday be
> >> sufficient
> >> > > extra
> >> > > >> time, so as not to slow down the anticipated release schedule too
> >> > much?
> >> > > >>
> >> > > >
> >> > > > Yes, I will probably have time to review it by Monday. But the
> >> > > > review-time concern is separate from the concern about which
> version
> >> > > > this should go into.
> >> > > >
> >> > >
> >> > > Reviewing this now... though I still think it shoudl target
> 0.20.206,
> >> > > not 0.20.205.1.
> >> > >
> >> > > -Todd
> >> > >
> >> > > >
> >> > > >>
> >> > > >> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]>
> >> > wrote:
> >> > > >>
> >> > > >>> Hey guys,
> >> > > >>>
> >> > > >>> HDFS-2246 is not a fix, it's a non-trivial performance
> >> optimization.
> >> > > >>> The roadmap page is pretty clear..  "Point releases are made to
> fix
> >> > > >>> critical bugs. They do not introduce new features or make other
> >> > > >>> improvements other than fixing bugs".
> >> > > >>>
> >> > > >>> I'm not opposed to the change, I'm just pointing out that we
> agreed
> >> > to
> >> > > >>> develop trunk first, and we agreed to follow the release
> policies
> >> for
> >> > > >>> the sustaining branch. I don't see why we can't honor those
> >

Hi all,
as noted in the previous thread, I apologize that I was not able to meet
the Nov 11 deadline.
I now intend to cut the 0.20.205.1 / 1.0.0 release candidate this weekend,
no sooner than Sat 26 Nov 8:00am PST.

Thank you,
--Matt

On Fri, Nov 4, 2011 at 5:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:

> Hi all,
> I propose to make a 0.20.205.1 candidate soon, with the following sets of
> patches:
>
>    - deficiencies in HBase support, pointed out by the HBase team and
>    others
>    - deficiencies in webhdfs support on secure clusters
>    - a couple last-minute fixes submitted to branch-0.20-security-205
>    that were too late to be included in 205.0
>
> If you would like other patches included, and you feel it is appropriate
> to have them in 205.1 rather than waiting for 206.0, please declare them by
> setting the "Target Versions" field in their Jiras, and they will receive
> due consideration, assuming that the proposed patch is actually
> contributed, tested, reviewed, approved, and committed
> to branch-0.20-security-205 by the freeze date :-)
>
> I would like to make the rc0 candidate next Friday, so I propose to
> declare 205.1 code freeze at noon, PST, Friday 11 Nov.  If this is a
> problem for anyone, please let me know.
>
> Thank you, and best regards,
> --Matt (Release Manager)
>
>
>

Hi Roman,
Do you have a proposed patch?  If so I would be happy to include it.

Thanks,
--Matt

On Wed, Nov 23, 2011 at 5:14 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:

> Hi Matt,
>
> quick question: any reason we are ignoring multifilewc from hadoop
> examples?
>   https://issues.apache.org/jira/browse/MAPREDUCE-3319
>
> would be nice to fix it for 1.0 of Hadoop. Or at least disable.
>
> Thanks,
> Roman.
>
> On Wed, Nov 23, 2011 at 2:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> > I really want this in 0.20.205.1, which will be Hadoop 1.0.0, because of
> > its importance for
> > good support of HBase.
> >
> > Jitendra, please merge it to branch-0.20-security-205.
> >
> > --Matt (wearing my Apache release manager hat)
> >
> >
> > On Wed, Nov 23, 2011 at 11:14 AM, Suresh Srinivas <
> [EMAIL PROTECTED]>wrote:
> >
> >> +1 for Jitendra's proposal.
> >>
> >> Additionally, most of the core of the code that this patch is based on
> has
> >> been tested and deployed in clusters at TrendMicro and Facebook.
> >>
> >> On Wed, Nov 23, 2011 at 11:10 AM, Jitendra Pandey
> >> <[EMAIL PROTECTED]>wrote:
> >>
> >> > The trunk, 206 patches for HDFS-2246 have been committed. I think it
> >> makes
> >> > sense to commit it to 205.1 as well for following reasons (most of it
> has
> >> > already been mentioned)
> >> > a) We intended this patch for 205, but couldn't finish in time. Now
> that
> >> > 205.1 branch is still not cut, we could get this in.
> >> > b) This is not a very risky change. Most of it is new code and will be
> >> > disabled by default the feature will be disabled.
> >> > c) The performance benefits are very good, as reported by Todd on the
> >> jira.
> >> > Hbase installations will significantly benefit from it.
> >> >
> >> > On Mon, Nov 14, 2011 at 1:44 PM, Todd Lipcon <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > On Fri, Nov 11, 2011 at 2:29 PM, Todd Lipcon <[EMAIL PROTECTED]>
> >> wrote:
> >> > > > On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <
> [EMAIL PROTECTED]>
> >> > > wrote:
> >> > > >
> >> > > >>
> >> > > >> Also, I believe in the HDFS-2246 Jira, Todd requested extra time
> to
> >> > > review,
> >> > > >> due to commitments at Hadoop World.  Todd, would Monday be
> >> sufficient
> >> > > extra
> >> > > >> time, so as not to slow down the anticipated release schedule too
> >> > much?
> >> > > >>
> >> > > >
> >> > > > Yes, I will probably have time to review it by Monday. But the
> >> > > > review-time concern is separate from the concern about which
> version
> >> > > > this should go into.
> >> > > >
> >> > >
> >> > > Reviewing this now... though I still think it shoudl target
> 0.20.206,
> >> > > not 0.20.205.1.
> >> > >
> >> > > -Todd
> >> > >
> >> > > >
> >> > > >>
> >> > > >> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]>
> >> > wrote:
> >> > > >>
> >> > > >>> Hey guys,
> >> > > >>>
> >> > > >>> HDFS-2246 is not a fix, it's a non-trivial performance
> >> optimization.
> >> > > >>> The roadmap page is pretty clear..  "Point releases are made to
> fix
> >> > > >>> critical bugs. They do not introduce new features or make other
> >> > > >>> improvements other than fixing bugs".
> >> > > >>>
> >> > > >>> I'm not opposed to the change, I'm just pointing out that we
> agreed
> >> > to
> >> > > >>> develop trunk first, and we agreed to follow the release
> policies
> >> for
> >> > > >>> the sustaining branch. I don't see why we can't honor those
> >> > > >>> agreements, ie why not post a patch for trunk first and then
> >> backport
> >> > > >>> it to 206? Reasonable?
> >> > > >>>
> >> > > >>> Thanks,
> >> > > >>> Eli
> >> > > >>>
> >> > > >>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <
> >> > > [EMAIL PROTECTED]>
> >> > > >>> wrote:
> >> > > >>> > Eli,
> >> > > >>> >
> >> > > >>> > As Jitendra indicated in the jira, this was originally
> supposed
> >> to
> >> > be
> >> > > >>> part
> >> > > >>> > of 0.205. Due to time crunc, we could not get this done in
> 0.205.
> >> > > This

On Nov 23, 2011, at 5:32 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:

> Hi Matt,
>
> quick question: any reason we are ignoring multifilewc from hadoop examples?
>

Roman - It isn't marked as a blocker nor is there a patch.

Can you provide one?

Arun

> https://issues.apache.org/jira/browse/MAPREDUCE-3319
>
> would be nice to fix it for 1.0 of Hadoop. Or at least disable.
>
> Thanks,
> Roman.
>
> On Wed, Nov 23, 2011 at 2:56 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
>> I really want this in 0.20.205.1, which will be Hadoop 1.0.0, because of
>> its importance for
>> good support of HBase.
>>
>> Jitendra, please merge it to branch-0.20-security-205.
>>
>> --Matt (wearing my Apache release manager hat)
>>
>>
>> On Wed, Nov 23, 2011 at 11:14 AM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:
>>
>>> +1 for Jitendra's proposal.
>>>
>>> Additionally, most of the core of the code that this patch is based on has
>>> been tested and deployed in clusters at TrendMicro and Facebook.
>>>
>>> On Wed, Nov 23, 2011 at 11:10 AM, Jitendra Pandey
>>> <[EMAIL PROTECTED]>wrote:
>>>
>>>> The trunk, 206 patches for HDFS-2246 have been committed. I think it
>>> makes
>>>> sense to commit it to 205.1 as well for following reasons (most of it has
>>>> already been mentioned)
>>>> a) We intended this patch for 205, but couldn't finish in time. Now that
>>>> 205.1 branch is still not cut, we could get this in.
>>>> b) This is not a very risky change. Most of it is new code and will be
>>>> disabled by default the feature will be disabled.
>>>> c) The performance benefits are very good, as reported by Todd on the
>>> jira.
>>>> Hbase installations will significantly benefit from it.
>>>>
>>>> On Mon, Nov 14, 2011 at 1:44 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> On Fri, Nov 11, 2011 at 2:29 PM, Todd Lipcon <[EMAIL PROTECTED]>
>>> wrote:
>>>>>> On Fri, Nov 11, 2011 at 1:29 AM, Matt Foley <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Also, I believe in the HDFS-2246 Jira, Todd requested extra time to
>>>>> review,
>>>>>>> due to commitments at Hadoop World.  Todd, would Monday be
>>> sufficient
>>>>> extra
>>>>>>> time, so as not to slow down the anticipated release schedule too
>>>> much?
>>>>>>>
>>>>>>
>>>>>> Yes, I will probably have time to review it by Monday. But the
>>>>>> review-time concern is separate from the concern about which version
>>>>>> this should go into.
>>>>>>
>>>>>
>>>>> Reviewing this now... though I still think it shoudl target 0.20.206,
>>>>> not 0.20.205.1.
>>>>>
>>>>> -Todd
>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 10, 2011 at 10:31 PM, Eli Collins <[EMAIL PROTECTED]>
>>>> wrote:
>>>>>>>
>>>>>>>> Hey guys,
>>>>>>>>
>>>>>>>> HDFS-2246 is not a fix, it's a non-trivial performance
>>> optimization.
>>>>>>>> The roadmap page is pretty clear..  "Point releases are made to fix
>>>>>>>> critical bugs. They do not introduce new features or make other
>>>>>>>> improvements other than fixing bugs".
>>>>>>>>
>>>>>>>> I'm not opposed to the change, I'm just pointing out that we agreed
>>>> to
>>>>>>>> develop trunk first, and we agreed to follow the release policies
>>> for
>>>>>>>> the sustaining branch. I don't see why we can't honor those
>>>>>>>> agreements, ie why not post a patch for trunk first and then
>>> backport
>>>>>>>> it to 206? Reasonable?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Eli
>>>>>>>>
>>>>>>>> On Thu, Nov 10, 2011 at 9:58 PM, Suresh Srinivas <
>>>>> [EMAIL PROTECTED]>
>>>>>>>> wrote:
>>>>>>>>> Eli,
>>>>>>>>>
>>>>>>>>> As Jitendra indicated in the jira, this was originally supposed
>>> to
>>>> be
>>>>>>>> part
>>>>>>>>> of 0.205. Due to time crunc, we could not get this done in 0.205.
>>>>> This
>>>>>>>> can
>>>>>>>>> be turned off by a flag and only can be enabled by users who want
>>>> to
>>>>> use
>>>>>>>>> the functionality. Given that, I feel it is okay to go into
>>>> 0.205.1.
>>>>>>>>>
>>>>>>>>> I agree it would be good to have a trunk patch for this and make
>>> it
>>>>> part
>>>>

On Nov 23, 2011, at 5:35 PM, Arun Murthy wrote:

> On Nov 23, 2011, at 5:32 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>
>> Hi Matt,
>>
>> quick question: any reason we are ignoring multifilewc from hadoop examples?
>>
>
> Roman - It isn't marked as a blocker nor is there a patch.
>

Also, another way to get the RM's attention is to mark the 'Target Version' along with the patch.

Arun

On Wed, Nov 23, 2011 at 5:40 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> Also, another way to get the RM's attention is to mark the 'Target Version' along with the patch.

Would be happy to do so. Wasn't sure what the proper protocol is.
Sorry about that.
As for the patch -- I'll add JIRA comments.

Thanks,
Roman.