Hi Matt,

My colleagues with experience running 0.20.203 (as well as many previous releases of Hadoop) in Yahoo!'s production environment are requesting that the following items be included as high priority sustaining improvements in 0.20.205.  Rather than contributors sending in several separate requests to this mailing list, this email aggregates contributions from the following individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne, John George.

Recommendations and suggestions for this list of jiras came from folks with significant experience working with large scale Hadoop clusters within Yahoo! production environments, including Service Engineering teams, Quality Engineering teams, Solutions Engineering teams, and Development teams.

Notes on the items listed below:

 *   All the Jiras listed with the exception of HADOOP-7510, MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to 0.20-security. These remaining four jiras are in-progress and should wrap up over the next few days.
 *   All of the jiras listed have been fixed in trunk with the following exceptions:
    *   The four jiras listed above which are still being worked
    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed MAPREDUCE-2723 to make sure this is handled correctly in yarn
    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration of MR279

Thank you for considering this list of Jiras for inclusion in 0.20.205.
Nathan Roberts

===
MAPREDUCE-2489 - Jobsplits with random hostnames can make the queue unusable
Justification: A broken job that is issuing random hostnames to the job tracker can hang up a queue and severely impact the performance of the job tracker.
Risk: Low. Change involves a simple check for obviously malformed hostnames.

MAPREDUCE-2852 - Remove YDH Bug 2854624 from  code comments
Justification: Comment change only
Risk: Low

HADOOP-7472 - RPC client should deal with the IP address changes
Justification: If the IP address of a namenode is changed, all clients must be restarted. This can be very expensive and difficult to execute when many of the clients are not within the cluster-proper. e.g. distcp
Risk: Low. If an address change is suspected, the code now performs an additional lookup and updates the address. Does not affect normal path.

MAPREDUCE-2729 - Reducers are always counted having pending tasks even if they can't be scheduled yet because not enough of their mappers have completed
Justification: reducer slots are not being properly allocated when reducers are waiting on map tasks to finish, causing situations where a queue can be significantly under utilized. In grids where queues are configured with relatively tight constraints, this can result in substantial throughput degradation when this condition arises.
Risk: Medium/Low. No change to the scheduler can be taken lightly so in those terms it's medium. However, the change itself is straighforward and the experts agree it was a bug.

MAPREDUCE-2705 - tasks localized and launched serially by TaskLauncher - causing other tasks to be delayed
Justification: Large localization processes lock up task launcher for potentially very long periods of time. This can result in significant delays for other tasks assigned to the same compute node.
Risk: Low. Localization is performed in a separate thread but overall flow for a particular task remains unchanged.

MAPREDUCE-2651 - Race condition in Linux Task Controller for job log directory creation
Justification: Tasks can fail because of a race to create the job log directory.
Risk: Low. Deals with EEXIST more consistently.

MAPREDUCE-2650 - back-port MAPREDUCE-2238 to 0.20-security
Justification: Permission handling within localization causes races and can leave directories with broken permissions. Adversely affects test reproducibility.
Risk: Low. Fix has been in trunk and 22 for several months.

MAPREDUCE-2621 - TestCapacityScheduler fails with Queue q1 does not exist
Justification: Hudson unit test failures
Risk: Low. Changes just create an explicit association between the QueueManager and JT.

MAPREDUCE-2494 - Make the distributed cache delete entires using LRU priority
Justification: Some regularly recurring jobs require large distributed cache contents. The current scheme deletes these contents when the distributed cache fills up. The penalty for localizing this type of job is a recurring penalty.
Risk: Low/Medium - Currently eviction is all or nothing. This change just orders the eviction and doesn't do it all at once. All of the races dealing with eviction were already dealt with in the code so no additional risk from that standpoint.

MAPREDUCE-2324 - Job should fail if a reduce task can't be scheduled anywhere
Justification: Jobs can get stuck in limbo emitting tons of messages to the logs about not being able to schedule the reduce. It's best to either just attempt the reduce and let it fail, or  put in more sophisticated logic to attempt to fail these jobs before attempting the reduce at all.
Risk: Low. Change now just removes the check which tried to prevent this. So, the job will be attempted and will just fail through the normal course.

MAPREDUCE-2187 - map tasks timeout during sorting
Justification: No progress is reported during merge sort so if this phase is takes too long, the tasks can timeout and fail.
Risk: Low. Adds new progress report point during merge sort.

HDFS-2202 - Changes to balancer bandwidth should not require datanode restart.
Justification: There are times when operations needs to either speed up or slow down the balancer bandwidth. The system should support doing so without restarting the datanodes.
Risk: Low/Medium - If feature is not used, code paths are the same.

HDFS-1836 - Thousand of CLOSE_WAIT socket
Justification: Clients can chew up socket connect

Thank you, Nathan.  And thanks for combining these requests, it's much
easier to review a single list than 10 or more emails.

Does anyone else have requests for items to include in 0.20.205 - either
already submitted to 0.20-security, or pending work?

Thanks,
--Matt

On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <[EMAIL PROTECTED]>wrote:

> Hi Matt,
>
> My colleagues with experience running 0.20.203 (as well as many previous
> releases of Hadoop) in Yahoo!'s production environment are requesting that
> the following items be included as high priority sustaining improvements in
> 0.20.205.  Rather than contributors sending in several separate requests to
> this mailing list, this email aggregates contributions from the following
> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
> John George.
>
> Recommendations and suggestions for this list of jiras came from folks with
> significant experience working with large scale Hadoop clusters within
> Yahoo! production environments, including Service Engineering teams, Quality
> Engineering teams, Solutions Engineering teams, and Development teams.
>
> Notes on the items listed below:
>
>  *   All the Jiras listed with the exception of HADOOP-7510,
> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
> 0.20-security. These remaining four jiras are in-progress and should wrap up
> over the next few days.
>  *   All of the jiras listed have been fixed in trunk with the following
> exceptions:
>    *   The four jiras listed above which are still being worked
>    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
> of MR279
>
> Thank you for considering this list of Jiras for inclusion in 0.20.205.
> Nathan Roberts
>
> ===> [list elided, please see archives at http://tinyurl.com/3k537hs ]

Hi Matt,

IMHO MAPREDUCE-2610 should be included in 205 as well.
We applied it to our branch because our users had problems with getting queue information.

Thanks,

Joep
________________________________________
From: Matt Foley [[EMAIL PROTECTED]]
Sent: Wednesday, August 31, 2011 11:16 AM
To: [EMAIL PROTECTED]
Subject: Re: Content request for 0.20.205 Sustaining Release

Thank you, Nathan.  And thanks for combining these requests, it's much
easier to review a single list than 10 or more emails.

Does anyone else have requests for items to include in 0.20.205 - either
already submitted to 0.20-security, or pending work?

Thanks,
--Matt

On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <[EMAIL PROTECTED]>wrote:

> Hi Matt,
>
> My colleagues with experience running 0.20.203 (as well as many previous
> releases of Hadoop) in Yahoo!'s production environment are requesting that
> the following items be included as high priority sustaining improvements in
> 0.20.205.  Rather than contributors sending in several separate requests to
> this mailing list, this email aggregates contributions from the following
> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
> John George.
>
> Recommendations and suggestions for this list of jiras came from folks with
> significant experience working with large scale Hadoop clusters within
> Yahoo! production environments, including Service Engineering teams, Quality
> Engineering teams, Solutions Engineering teams, and Development teams.
>
> Notes on the items listed below:
>
>  *   All the Jiras listed with the exception of HADOOP-7510,
> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
> 0.20-security. These remaining four jiras are in-progress and should wrap up
> over the next few days.
>  *   All of the jiras listed have been fixed in trunk with the following
> exceptions:
>    *   The four jiras listed above which are still being worked
>    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
> of MR279
>
> Thank you for considering this list of Jiras for inclusion in 0.20.205.
> Nathan Roberts
>
> ===> [list elided, please see archives at http://tinyurl.com/3k537hs ]

Could you please add MAPREDUCE-2779 to the list as well?
This one has a patch as well.

Thanks,

Joep
________________________________________
From: Rottinghuis, Joep [[EMAIL PROTECTED]]
Sent: Thursday, September 01, 2011 9:04 AM
To: [EMAIL PROTECTED]
Subject: RE: Content request for 0.20.205 Sustaining Release

Hi Matt,

IMHO MAPREDUCE-2610 should be included in 205 as well.
We applied it to our branch because our users had problems with getting queue information.

Thanks,

Joep
________________________________________
From: Matt Foley [[EMAIL PROTECTED]]
Sent: Wednesday, August 31, 2011 11:16 AM
To: [EMAIL PROTECTED]
Subject: Re: Content request for 0.20.205 Sustaining Release

Thank you, Nathan.  And thanks for combining these requests, it's much
easier to review a single list than 10 or more emails.

Does anyone else have requests for items to include in 0.20.205 - either
already submitted to 0.20-security, or pending work?

Thanks,
--Matt

On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <[EMAIL PROTECTED]>wrote:

> Hi Matt,
>
> My colleagues with experience running 0.20.203 (as well as many previous
> releases of Hadoop) in Yahoo!'s production environment are requesting that
> the following items be included as high priority sustaining improvements in
> 0.20.205.  Rather than contributors sending in several separate requests to
> this mailing list, this email aggregates contributions from the following
> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
> John George.
>
> Recommendations and suggestions for this list of jiras came from folks with
> significant experience working with large scale Hadoop clusters within
> Yahoo! production environments, including Service Engineering teams, Quality
> Engineering teams, Solutions Engineering teams, and Development teams.
>
> Notes on the items listed below:
>
>  *   All the Jiras listed with the exception of HADOOP-7510,
> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
> 0.20-security. These remaining four jiras are in-progress and should wrap up
> over the next few days.
>  *   All of the jiras listed have been fixed in trunk with the following
> exceptions:
>    *   The four jiras listed above which are still being worked
>    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
> of MR279
>
> Thank you for considering this list of Jiras for inclusion in 0.20.205.
> Nathan Roberts
>
> ===> [list elided, please see archives at http://tinyurl.com/3k537hs ]

Hi Matt,
I would like to bring HADOOP-7602 to your attention. I just filed it today
and have a patch uploaded, but I think it should go in to 205.
Thank you for considering.

Regads,
John George
On 9/1/11 11:08 AM, "Rottinghuis, Joep" <[EMAIL PROTECTED]> wrote:

> Could you please add MAPREDUCE-2779 to the list as well?
> This one has a patch as well.
>
> Thanks,
>
> Joep
> ________________________________________
> From: Rottinghuis, Joep [[EMAIL PROTECTED]]
> Sent: Thursday, September 01, 2011 9:04 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Content request for 0.20.205 Sustaining Release
>
> Hi Matt,
>
> IMHO MAPREDUCE-2610 should be included in 205 as well.
> We applied it to our branch because our users had problems with getting queue
> information.
>
> Thanks,
>
> Joep
> ________________________________________
> From: Matt Foley [[EMAIL PROTECTED]]
> Sent: Wednesday, August 31, 2011 11:16 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Content request for 0.20.205 Sustaining Release
>
> Thank you, Nathan.  And thanks for combining these requests, it's much
> easier to review a single list than 10 or more emails.
>
> Does anyone else have requests for items to include in 0.20.205 - either
> already submitted to 0.20-security, or pending work?
>
> Thanks,
> --Matt
>
> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <[EMAIL PROTECTED]>wrote:
>
>> Hi Matt,
>>
>> My colleagues with experience running 0.20.203 (as well as many previous
>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>> the following items be included as high priority sustaining improvements in
>> 0.20.205.  Rather than contributors sending in several separate requests to
>> this mailing list, this email aggregates contributions from the following
>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>> John George.
>>
>> Recommendations and suggestions for this list of jiras came from folks with
>> significant experience working with large scale Hadoop clusters within
>> Yahoo! production environments, including Service Engineering teams, Quality
>> Engineering teams, Solutions Engineering teams, and Development teams.
>>
>> Notes on the items listed below:
>>
>>  *   All the Jiras listed with the exception of HADOOP-7510,
>> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
>> 0.20-security. These remaining four jiras are in-progress and should wrap up
>> over the next few days.
>>  *   All of the jiras listed have been fixed in trunk with the following
>> exceptions:
>>    *   The four jiras listed above which are still being worked
>>    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>>    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
>> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>>    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
>> of MR279
>>
>> Thank you for considering this list of Jiras for inclusion in 0.20.205.
>> Nathan Roberts
>>
>> ===>> [list elided, please see archives at http://tinyurl.com/3k537hs ]

Thanks for the input guys - I'll look at MAPREDUCE-2601,MAPREDUCE-2779 & HADOOP-7602.

Arun

On Sep 1, 2011, at 11:02 AM, John George wrote:

> Hi Matt,
> I would like to bring HADOOP-7602 to your attention. I just filed it today
> and have a patch uploaded, but I think it should go in to 205.
> Thank you for considering.
>
> Regads,
> John George
>
>
> On 9/1/11 11:08 AM, "Rottinghuis, Joep" <[EMAIL PROTECTED]> wrote:
>
>> Could you please add MAPREDUCE-2779 to the list as well?
>> This one has a patch as well.
>>
>> Thanks,
>>
>> Joep
>> ________________________________________
>> From: Rottinghuis, Joep [[EMAIL PROTECTED]]
>> Sent: Thursday, September 01, 2011 9:04 AM
>> To: [EMAIL PROTECTED]
>> Subject: RE: Content request for 0.20.205 Sustaining Release
>>
>> Hi Matt,
>>
>> IMHO MAPREDUCE-2610 should be included in 205 as well.
>> We applied it to our branch because our users had problems with getting queue
>> information.
>>
>> Thanks,
>>
>> Joep
>> ________________________________________
>> From: Matt Foley [[EMAIL PROTECTED]]
>> Sent: Wednesday, August 31, 2011 11:16 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Content request for 0.20.205 Sustaining Release
>>
>> Thank you, Nathan.  And thanks for combining these requests, it's much
>> easier to review a single list than 10 or more emails.
>>
>> Does anyone else have requests for items to include in 0.20.205 - either
>> already submitted to 0.20-security, or pending work?
>>
>> Thanks,
>> --Matt
>>
>> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <[EMAIL PROTECTED]>wrote:
>>
>>> Hi Matt,
>>>
>>> My colleagues with experience running 0.20.203 (as well as many previous
>>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>>> the following items be included as high priority sustaining improvements in
>>> 0.20.205.  Rather than contributors sending in several separate requests to
>>> this mailing list, this email aggregates contributions from the following
>>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>>> John George.
>>>
>>> Recommendations and suggestions for this list of jiras came from folks with
>>> significant experience working with large scale Hadoop clusters within
>>> Yahoo! production environments, including Service Engineering teams, Quality
>>> Engineering teams, Solutions Engineering teams, and Development teams.
>>>
>>> Notes on the items listed below:
>>>
>>> *   All the Jiras listed with the exception of HADOOP-7510,
>>> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
>>> 0.20-security. These remaining four jiras are in-progress and should wrap up
>>> over the next few days.
>>> *   All of the jiras listed have been fixed in trunk with the following
>>> exceptions:
>>>   *   The four jiras listed above which are still being worked
>>>   *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>>>   *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
>>> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>>>   *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
>>> of MR279
>>>
>>> Thank you for considering this list of Jiras for inclusion in 0.20.205.
>>> Nathan Roberts
>>>
>>> ===>>> [list elided, please see archives at http://tinyurl.com/3k537hs ]
>

Hi Matt,

I'd like to propose backporting some of the new fair scheduler features that have been out for almost 2 years now (in the 0.21 branch and trunk) into 0.20.205, especially https://issues.apache.org/jira/browse/MAPREDUCE-551 and https://issues.apache.org/jira/browse/MAPREDUCE-706. I will try to put together a patch for this. It should be a matter of taking the trunk fairscheduler directory and just tweaking anything that might be required to compile on 0.20.205; I don't think there are other major differences.

Matei

On Sep 1, 2011, at 11:48 AM, Arun C Murthy wrote:

> Thanks for the input guys - I'll look at MAPREDUCE-2601,MAPREDUCE-2779 & HADOOP-7602.
>
> Arun
>
> On Sep 1, 2011, at 11:02 AM, John George wrote:
>
>> Hi Matt,
>> I would like to bring HADOOP-7602 to your attention. I just filed it today
>> and have a patch uploaded, but I think it should go in to 205.
>> Thank you for considering.
>>
>> Regads,
>> John George
>>
>>
>> On 9/1/11 11:08 AM, "Rottinghuis, Joep" <[EMAIL PROTECTED]> wrote:
>>
>>> Could you please add MAPREDUCE-2779 to the list as well?
>>> This one has a patch as well.
>>>
>>> Thanks,
>>>
>>> Joep
>>> ________________________________________
>>> From: Rottinghuis, Joep [[EMAIL PROTECTED]]
>>> Sent: Thursday, September 01, 2011 9:04 AM
>>> To: [EMAIL PROTECTED]
>>> Subject: RE: Content request for 0.20.205 Sustaining Release
>>>
>>> Hi Matt,
>>>
>>> IMHO MAPREDUCE-2610 should be included in 205 as well.
>>> We applied it to our branch because our users had problems with getting queue
>>> information.
>>>
>>> Thanks,
>>>
>>> Joep
>>> ________________________________________
>>> From: Matt Foley [[EMAIL PROTECTED]]
>>> Sent: Wednesday, August 31, 2011 11:16 AM
>>> To: [EMAIL PROTECTED]
>>> Subject: Re: Content request for 0.20.205 Sustaining Release
>>>
>>> Thank you, Nathan.  And thanks for combining these requests, it's much
>>> easier to review a single list than 10 or more emails.
>>>
>>> Does anyone else have requests for items to include in 0.20.205 - either
>>> already submitted to 0.20-security, or pending work?
>>>
>>> Thanks,
>>> --Matt
>>>
>>> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi Matt,
>>>>
>>>> My colleagues with experience running 0.20.203 (as well as many previous
>>>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>>>> the following items be included as high priority sustaining improvements in
>>>> 0.20.205.  Rather than contributors sending in several separate requests to
>>>> this mailing list, this email aggregates contributions from the following
>>>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>>>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>>>> John George.
>>>>
>>>> Recommendations and suggestions for this list of jiras came from folks with
>>>> significant experience working with large scale Hadoop clusters within
>>>> Yahoo! production environments, including Service Engineering teams, Quality
>>>> Engineering teams, Solutions Engineering teams, and Development teams.
>>>>
>>>> Notes on the items listed below:
>>>>
>>>> *   All the Jiras listed with the exception of HADOOP-7510,
>>>> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
>>>> 0.20-security. These remaining four jiras are in-progress and should wrap up
>>>> over the next few days.
>>>> *   All of the jiras listed have been fixed in trunk with the following
>>>> exceptions:
>>>>  *   The four jiras listed above which are still being worked
>>>>  *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>>>>  *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
>>>> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>>>>  *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
>>>> of MR279
>>>>
>>>> Thank you for considering this list of Jiras for inclusion in 0.20.205.

Agree, +1.

It will be good to ensure we get a more modern version of
FairScheduler for interested folks.

Arun

Sent from my iPhone

On Sep 2, 2011, at 5:55 PM, Matei Zaharia <[EMAIL PROTECTED]> wrote:

> Hi Matt,
>
> I'd like to propose backporting some of the new fair scheduler features that have been out for almost 2 years now (in the 0.21 branch and trunk) into 0.20.205, especially https://issues.apache.org/jira/browse/MAPREDUCE-551 and https://issues.apache.org/jira/browse/MAPREDUCE-706. I will try to put together a patch for this. It should be a matter of taking the trunk fairscheduler directory and just tweaking anything that might be required to compile on 0.20.205; I don't think there are other major differences.
>
> Matei
>
> On Sep 1, 2011, at 11:48 AM, Arun C Murthy wrote:
>
>> Thanks for the input guys - I'll look at MAPREDUCE-2601,MAPREDUCE-2779 & HADOOP-7602.
>>
>> Arun
>>
>> On Sep 1, 2011, at 11:02 AM, John George wrote:
>>
>>> Hi Matt,
>>> I would like to bring HADOOP-7602 to your attention. I just filed it today
>>> and have a patch uploaded, but I think it should go in to 205.
>>> Thank you for considering.
>>>
>>> Regads,
>>> John George
>>>
>>>
>>> On 9/1/11 11:08 AM, "Rottinghuis, Joep" <[EMAIL PROTECTED]> wrote:
>>>
>>>> Could you please add MAPREDUCE-2779 to the list as well?
>>>> This one has a patch as well.
>>>>
>>>> Thanks,
>>>>
>>>> Joep
>>>> ________________________________________
>>>> From: Rottinghuis, Joep [[EMAIL PROTECTED]]
>>>> Sent: Thursday, September 01, 2011 9:04 AM
>>>> To: [EMAIL PROTECTED]
>>>> Subject: RE: Content request for 0.20.205 Sustaining Release
>>>>
>>>> Hi Matt,
>>>>
>>>> IMHO MAPREDUCE-2610 should be included in 205 as well.
>>>> We applied it to our branch because our users had problems with getting queue
>>>> information.
>>>>
>>>> Thanks,
>>>>
>>>> Joep
>>>> ________________________________________
>>>> From: Matt Foley [[EMAIL PROTECTED]]
>>>> Sent: Wednesday, August 31, 2011 11:16 AM
>>>> To: [EMAIL PROTECTED]
>>>> Subject: Re: Content request for 0.20.205 Sustaining Release
>>>>
>>>> Thank you, Nathan.  And thanks for combining these requests, it's much
>>>> easier to review a single list than 10 or more emails.
>>>>
>>>> Does anyone else have requests for items to include in 0.20.205 - either
>>>> already submitted to 0.20-security, or pending work?
>>>>
>>>> Thanks,
>>>> --Matt
>>>>
>>>> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi Matt,
>>>>>
>>>>> My colleagues with experience running 0.20.203 (as well as many previous
>>>>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>>>>> the following items be included as high priority sustaining improvements in
>>>>> 0.20.205.  Rather than contributors sending in several separate requests to
>>>>> this mailing list, this email aggregates contributions from the following
>>>>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>>>>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>>>>> John George.
>>>>>
>>>>> Recommendations and suggestions for this list of jiras came from folks with
>>>>> significant experience working with large scale Hadoop clusters within
>>>>> Yahoo! production environments, including Service Engineering teams, Quality
>>>>> Engineering teams, Solutions Engineering teams, and Development teams.
>>>>>
>>>>> Notes on the items listed below:
>>>>>
>>>>> *   All the Jiras listed with the exception of HADOOP-7510,
>>>>> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
>>>>> 0.20-security. These remaining four jiras are in-progress and should wrap up
>>>>> over the next few days.
>>>>> *   All of the jiras listed have been fixed in trunk with the following
>>>>> exceptions:
>>>>> *   The four jiras listed above which are still being worked
>>>>> *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>>>>> *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed

Matt, Others,

Is the expectation that these fixes go into other release branches too (including 0.22, 0.23) if applicable?  

If not, my concern is that 0.22 is regressing further from the popular Apache release and I'm inclined to abandon 0.22. Thoughts?

Cheers,
Nige

On Sep 2, 2011, at 6:40 PM, Arun Murthy <[EMAIL PROTECTED]> wrote:

> Agree, +1.
>
> It will be good to ensure we get a more modern version of
> FairScheduler for interested folks.
>
> Arun
>
> Sent from my iPhone
>
> On Sep 2, 2011, at 5:55 PM, Matei Zaharia <[EMAIL PROTECTED]> wrote:
>
>> Hi Matt,
>>
>> I'd like to propose backporting some of the new fair scheduler features that have been out for almost 2 years now (in the 0.21 branch and trunk) into 0.20.205, especially https://issues.apache.org/jira/browse/MAPREDUCE-551 and https://issues.apache.org/jira/browse/MAPREDUCE-706. I will try to put together a patch for this. It should be a matter of taking the trunk fairscheduler directory and just tweaking anything that might be required to compile on 0.20.205; I don't think there are other major differences.
>>
>> Matei
>>
>> On Sep 1, 2011, at 11:48 AM, Arun C Murthy wrote:
>>
>>> Thanks for the input guys - I'll look at MAPREDUCE-2601,MAPREDUCE-2779 & HADOOP-7602.
>>>
>>> Arun
>>>
>>> On Sep 1, 2011, at 11:02 AM, John George wrote:
>>>
>>>> Hi Matt,
>>>> I would like to bring HADOOP-7602 to your attention. I just filed it today
>>>> and have a patch uploaded, but I think it should go in to 205.
>>>> Thank you for considering.
>>>>
>>>> Regads,
>>>> John George
>>>>
>>>>
>>>> On 9/1/11 11:08 AM, "Rottinghuis, Joep" <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Could you please add MAPREDUCE-2779 to the list as well?
>>>>> This one has a patch as well.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Joep
>>>>> ________________________________________
>>>>> From: Rottinghuis, Joep [[EMAIL PROTECTED]]
>>>>> Sent: Thursday, September 01, 2011 9:04 AM
>>>>> To: [EMAIL PROTECTED]
>>>>> Subject: RE: Content request for 0.20.205 Sustaining Release
>>>>>
>>>>> Hi Matt,
>>>>>
>>>>> IMHO MAPREDUCE-2610 should be included in 205 as well.
>>>>> We applied it to our branch because our users had problems with getting queue
>>>>> information.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Joep
>>>>> ________________________________________
>>>>> From: Matt Foley [[EMAIL PROTECTED]]
>>>>> Sent: Wednesday, August 31, 2011 11:16 AM
>>>>> To: [EMAIL PROTECTED]
>>>>> Subject: Re: Content request for 0.20.205 Sustaining Release
>>>>>
>>>>> Thank you, Nathan.  And thanks for combining these requests, it's much
>>>>> easier to review a single list than 10 or more emails.
>>>>>
>>>>> Does anyone else have requests for items to include in 0.20.205 - either
>>>>> already submitted to 0.20-security, or pending work?
>>>>>
>>>>> Thanks,
>>>>> --Matt
>>>>>
>>>>> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> Hi Matt,
>>>>>>
>>>>>> My colleagues with experience running 0.20.203 (as well as many previous
>>>>>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>>>>>> the following items be included as high priority sustaining improvements in
>>>>>> 0.20.205.  Rather than contributors sending in several separate requests to
>>>>>> this mailing list, this email aggregates contributions from the following
>>>>>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>>>>>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>>>>>> John George.
>>>>>>
>>>>>> Recommendations and suggestions for this list of jiras came from folks with
>>>>>> significant experience working with large scale Hadoop clusters within
>>>>>> Yahoo! production environments, including Service Engineering teams, Quality
>>>>>> Engineering teams, Solutions Engineering teams, and Development teams.
>>>>>>
>>>>>> Notes on the items listed below:
>>>>>>
>>>>>> *   All the Jiras listed with the exception of HADOOP-7510,