Hi Guys.

how are we going with the 0.22 release.

--
Ian Holsman
[EMAIL PROTECTED]
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

We are not afraid of the truth, in fact we plan on taking the truth out for a nice meal while we persuade it to adopt our views

On Dec 16, 2010, at 10:54 AM, Ian Holsman wrote:

> how are we going with the 0.22 release.

Progress has been blocked because HADOOP-6685 is still blocked. I feel  
that HADOOP-6685 is critical for Hadoop and don't see much point in  
working on a 0.22 release without it. In particular, HADOOP-6685  
removes the need for third party serialization libraries that work  
around the current limitations of the MapReduce framework, but  
splinter the user community.

If someone else wants to take over as release manager for 0.22 while  
we find a solution to make forward progress on HADOOP-6685, that would  
be fine.

-- Owen

On Dec 17, 2010, at 6:15 AM, Owen O'Malley wrote:

>
> On Dec 16, 2010, at 10:54 AM, Ian Holsman wrote:
>
>> how are we going with the 0.22 release.
>
> Progress has been blocked because HADOOP-6685 is still blocked. I feel that HADOOP-6685 is critical for Hadoop and don't see much point in working on a 0.22 release without it.

While it may be critical, I still don't understand why we couldn't release without it, and carry on the discussion.
Putting so much emphasis and pressure on this issue is unwarranted.

Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
There are now 4 separate companies who have released their own 0.20 version because of the delay on our releasing...
Delaying this release another 2-3 months for a single patch seems a bit extreme to me.

 

> In particular, HADOOP-6685 removes the need for third party serialization libraries that work around the current limitations of the MapReduce framework, but splinter the user community.
>
> If someone else wants to take over as release manager for 0.22 while we find a solution to make forward progress on HADOOP-6685, that would be fine.
>
> -- Owen

On Dec 16, 2010, at 12:26 PM, Ian Holsman wrote:

> While it may be critical, I still don't understand why we couldn't  
> release without it, and carry on the discussion.
> Putting so much emphasis and pressure on this issue is unwarranted.

It took months to stabilize 0.21 and it took a lot of Tom's time.  
H-6685 is important to me and needs to be addressed to move the  
project forward.

> Everyone who has discussed this patch has said it isn't critical to  
> hadoop, and It's holding up everything else 0.22 is going to bring.

I disagree that it isn't critical to Hadoop, but I'm not holding up  
0.22. I'm just not volunteering to spend my time working on it, if it  
doesn't have the features that I think it needs.

-- Owen

On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:

>> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
>
> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
>
> -- Owen
>

That's a fair point.. and this is a volunteer effort.
Do we have anybody else who is willing to be the release manager for 22 with 6685?

On Fri, Dec 17, 2010 at 10:10AM, Ian Holsman wrote:
>
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>
> >> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
> >
> > I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
> >
> > -- Owen
> >
>
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22 with 6685?

You meant to say "...to be the release manager for 22 without 6685?", didn't you?

Cos

On Dec 17, 2010, at 4:16 PM, Konstantin Boudnik wrote:
>
>
> You meant to say "...to be the release manager for 22 without 6685?", didn't you?

Yes..
It's been a long day ;-(

>
> Cos
>

Owen, Doug, Tom

Could you please formulate and reply to this email separately
     what would be an *ACCEPTABLE *resolution of
     HADOOP-6685 for *YOU *to move *0.22* forward.
Just trying to get something to work with to get us beyond the stagnation
point.

It could be "I want this patch in/out as is, final answer". Then we are
stuck.
But at least we will know there is no resolution to hope for anymore.
And we have to find other ways based on that fact.

It could be a zero-option plan - remove dependencies both for Avro and
ProtocolBuffers out into libraries, similar to schedulers.
Or something else.
Let's see if there is any common ground. If there is
we can further talk about implementation and in the mean time declare
the 0.22 freeze contingent on the completion of H-6685.

Thanks,
--Konstantin
On Thu, Dec 16, 2010 at 3:10 PM, Ian Holsman <[EMAIL PROTECTED]> wrote:

>
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>
> >> Everyone who has discussed this patch has said it isn't critical to
> hadoop, and It's holding up everything else 0.22 is going to bring.
> >
> > I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22.
> I'm just not volunteering to spend my time working on it, if it doesn't have
> the features that I think it needs.
> >
> > -- Owen
> >
>
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22
> with 6685?
>
>

On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
> It could be a zero-option plan - remove dependencies both for Avro and
> ProtocolBuffers out into libraries, similar to schedulers.

I'd be fine with removing Avro from the mapreduce user's classpath.
It's currently an unused option for RPC, and is used server-side on the
jobtracker.  It could be removed from the user classpath today without
loss of critical functionality.

Doug

On Fri, Dec 17, 2010 at 2:52 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
>
>> It could be a zero-option plan - remove dependencies both for Avro and
>> ProtocolBuffers out into libraries, similar to schedulers.
>>
>
> I'd be fine with removing Avro from the mapreduce user's classpath. It's
> currently an unused option for RPC, and is used server-side on the
> jobtracker.  It could be removed from the user classpath today without loss
> of critical functionality.
That would be non-trivial and no one has requested that. You haven't
answered Konstantin's question.

What do you consider the technical reasons for rejecting the patch as is?

-- Owen

My question is
    What would be an *ACCEPTABLE *resolution of HADOOP-6685 for *YOU *to
move *0.22* forward?
Asking Owen as the author of the patch, Doug and Tom as they vetoed it.

If I am asking the wrong question please let me know.
I am not proposing a particular solution, just trying to understand
1. if there is any
2. What is it if there is.

If, say removing dependency for Avro and updating the patch symmetrically
removing PB dependency is a solution, if there is a solution to SequenceFile
and byte array issues, then lets do it.
If not then lets face it.

Thanks,
--Konstantin

On Sun, Dec 19, 2010 at 11:27 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote:

> On Fri, Dec 17, 2010 at 2:52 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
>
> > On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
> >
> >> It could be a zero-option plan - remove dependencies both for Avro and
> >> ProtocolBuffers out into libraries, similar to schedulers.
> >>
> >
> > I'd be fine with removing Avro from the mapreduce user's classpath. It's
> > currently an unused option for RPC, and is used server-side on the
> > jobtracker.  It could be removed from the user classpath today without
> loss
> > of critical functionality.
>
>
> That would be non-trivial and no one has requested that. You haven't
> answered Konstantin's question.
>
> What do you consider the technical reasons for rejecting the patch as is?
>
> -- Owen
>

[Sending this again as my original post didn't make it to the list for
some reason. Can't see it in the moderation queue either.]

I don't personally see HADOOP-6685 as a blocker for a 0.22 release,
since there is a lot of value in there already that has not been
released yet, such as security. However, to get HADOOP-6685 resolved,
from my point of view the main thing to sort out are the modularity
concerns that I and others have raised, so that serializations are
pluggable and don't add potentially incompatible libraries onto the
user's classpath.

Thanks,
Tom

On Mon, Dec 20, 2010 at 11:39 AM, Konstantin Shvachko
<[EMAIL PROTECTED]> wrote:
> My question is
>    What would be an *ACCEPTABLE *resolution of HADOOP-6685 for *YOU *to
> move *0.22* forward?
> Asking Owen as the author of the patch, Doug and Tom as they vetoed it.
>
> If I am asking the wrong question please let me know.
> I am not proposing a particular solution, just trying to understand
> 1. if there is any
> 2. What is it if there is.
>
> If, say removing dependency for Avro and updating the patch symmetrically
> removing PB dependency is a solution, if there is a solution to SequenceFile
> and byte array issues, then lets do it.
> If not then lets face it.
>
> Thanks,
> --Konstantin
>
> On Sun, Dec 19, 2010 at 11:27 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
>
>> On Fri, Dec 17, 2010 at 2:52 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
>>
>> > On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
>> >
>> >> It could be a zero-option plan - remove dependencies both for Avro and
>> >> ProtocolBuffers out into libraries, similar to schedulers.
>> >>
>> >
>> > I'd be fine with removing Avro from the mapreduce user's classpath. It's
>> > currently an unused option for RPC, and is used server-side on the
>> > jobtracker.  It could be removed from the user classpath today without
>> loss
>> > of critical functionality.
>>
>>
>> That would be non-trivial and no one has requested that. You haven't
>> answered Konstantin's question.
>>
>> What do you consider the technical reasons for rejecting the patch as is?
>>
>> -- Owen
>>
>

On Mon, Dec 20, 2010 at 12:12 PM, Tom White <[EMAIL PROTECTED]> wrote:

> However, to get HADOOP-6685 resolved,
> from my point of view the main thing to sort out are the modularity
> concerns that I and others have raised, so that serializations are
> pluggable and don't add potentially incompatible libraries onto the
> user's classpath.
>

The serializers are already completely pluggable. If you want to propose a
plan for reducing the number of jars on the user's task classpath that might
be interesting. Clearly, it would be a separate jira, since it has nothing
to do with the serialization interface.

-- Owen

On Fri, Dec 17, 2010 at 2:34 PM, Konstantin Shvachko
<[EMAIL PROTECTED]> wrote:
> Owen, Doug, Tom
>
> Could you please formulate and reply to this email separately
>     what would be an *ACCEPTABLE *resolution of
>     HADOOP-6685 for *YOU *to move *0.22* forward.
> Just trying to get something to work with to get us beyond the stagnation
> point.
>
> It could be "I want this patch in/out as is, final answer". Then we are
> stuck.
> But at least we will know there is no resolution to hope for anymore.
> And we have to find other ways based on that fact.
>
> It could be a zero-option plan - remove dependencies both for Avro and
> ProtocolBuffers out into libraries, similar to schedulers.
> Or something else.
> Let's see if there is any common ground. If there is
> we can further talk about implementation and in the mean time declare
> the 0.22 freeze contingent on the completion of H-6685.

I don't personally see HADOOP-6685 as a blocker for a 0.22 release,
since there is a lot of value in there already that has not been
released yet, such as security. However, to get HADOOP-6685 resolved,
from my point of view the main thing to sort out are the modularity
concerns that I and others have raised, so that serializations are
pluggable and don't add potentially incompatible libraries onto the
user's classpath.

Thanks,
Tom

>
> Thanks,
> --Konstantin
>
>
> On Thu, Dec 16, 2010 at 3:10 PM, Ian Holsman <[EMAIL PROTECTED]> wrote:
>
>>
>> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>>
>> >> Everyone who has discussed this patch has said it isn't critical to
>> hadoop, and It's holding up everything else 0.22 is going to bring.
>> >
>> > I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22.
>> I'm just not volunteering to spend my time working on it, if it doesn't have
>> the features that I think it needs.
>> >
>> > -- Owen
>> >
>>
>> That's a fair point.. and this is a volunteer effort.
>> Do we have anybody else who is willing to be the release manager for 22
>> with 6685?
>>
>>
>

I think it's important we get more frequent releases back on track, so I'll volunteer to do it.  Its been a while since I was the release manager so I'll need help.

Looks like in Jira we currently have:
  26 MR blockers: http://tinyurl.com/23s5vzg
  17 HDFS blockers: http://tinyurl.com/27w82vl
  6 HADOOP blockers: http://tinyurl.com/2bmycl2
Now that we are past feature freeze (i.e. branch creation), not all of these are truly blockers.  I'll start cleaning up the list and ping folks after the holidays.

Cheers,
Nige

On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:

>
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>
>>> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
>>
>> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
>>
>> -- Owen
>>
>
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22 with 6685?
>

Thanks Nige.

On Dec 22, 2010, at 6:24 AM, Nigel Daley wrote:

> I think it's important we get more frequent releases back on track, so I'll volunteer to do it.  Its been a while since I was the release manager so I'll need help.
>
> Looks like in Jira we currently have:
>  26 MR blockers: http://tinyurl.com/23s5vzg
>  17 HDFS blockers: http://tinyurl.com/27w82vl
>  6 HADOOP blockers: http://tinyurl.com/2bmycl2
> Now that we are past feature freeze (i.e. branch creation), not all of these are truly blockers.  I'll start cleaning up the list and ping folks after the holidays.
>
> Cheers,
> Nige
>
> On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
>>
>> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>>
>>>> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
>>>
>>> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
>>>
>>> -- Owen
>>>
>>
>> That's a fair point.. and this is a volunteer effort.
>> Do we have anybody else who is willing to be the release manager for 22 with 6685?
>>
>

+1

On Tue, Dec 21, 2010 at 11:24, Nigel Daley <[EMAIL PROTECTED]> wrote:
> I think it's important we get more frequent releases back on track, so I'll volunteer to do it.  Its been a while since I was the release manager so I'll need help.
>
> Looks like in Jira we currently have:
>  26 MR blockers: http://tinyurl.com/23s5vzg
>  17 HDFS blockers: http://tinyurl.com/27w82vl
>  6 HADOOP blockers: http://tinyurl.com/2bmycl2
> Now that we are past feature freeze (i.e. branch creation), not all of these are truly blockers.  I'll start cleaning up the list and ping folks after the holidays.
>
> Cheers,
> Nige
>
> On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
>>
>> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>>
>>>> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
>>>
>>> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
>>>
>>> -- Owen
>>>
>>
>> That's a fair point.. and this is a volunteer effort.
>> Do we have anybody else who is willing to be the release manager for 22 with 6685?
>>
>
>

Not sure what state this issue is in, would be nice for 0.22

https://issues.apache.org/jira/browse/HADOOP-6941

I don't see it in the lists below.  It's a bit of a blocker running on
non-Sun (Oracle) JREs.
                                                                      
                                                                      
                                                                      
                                                                      
                                                                      
 Ken (K.N.)                                  2670 Queensview          
 Walker                                     Drive                      
                                                                      
 J9 Java                                     Ottawa, Ontario K2B      
 Class                                      8K1                        
 Libraries                                                            
 Lead                                                                  
                                                                      
 Java                                        Canada                    
 Technology                                                            
 Centre                                                                
                                                                      
 IBM                                                                  
 Software                                                              
 Group                                                                
                                                                      
 Phone:     +1-613-356-5021                                            
                                                                      
 e-mail:    [EMAIL PROTECTED]                                      
                                                                      
                                                                      
                                                                      
                                                                      
                                                                      
|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Nigel Daley <[EMAIL PROTECTED]>                                                                                                                      |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |[EMAIL PROTECTED]                                                                                                                         |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |2010/12/21 02:26 PM                                                                                                                               |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Re: Plans for the 0.22 Release                                                                                                                    |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|

I think it's important we get more frequent releases back on track, so I'll
volunteer to do it.  Its been a while since I was the release manager so
I'll need help.

Looks like in Jira we currently have:
  26 MR blockers: http://tinyurl.com/23s5vzg
  17 HDFS blockers: http://tinyurl.com/27w82vl
  6 HADOOP blockers: http://tinyurl.com/2bmycl2
Now that we are past feature freeze (i.e. branch creation), not all of
these are truly blockers.  I'll start cleaning up the list and ping folks
after the holidays.

Cheers,
Nige

On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:

hadoop, and It's holding up everything else 0.22 is going to bring.
0.22. I'm just not volunteering to spend my time working on it, if it
doesn't have the features that I think it needs.
with 6685?

This is great! Thanks, Nigel.

I haven't seen the feature freeze for 0.22 formally announced.
May be it worth confirming it now by setting the start day, as of yesterday?

--Konstantin

On Tue, Dec 21, 2010 at 11:24 AM, Nigel Daley <[EMAIL PROTECTED]> wrote:

> I think it's important we get more frequent releases back on track, so I'll
> volunteer to do it.  Its been a while since I was the release manager so
> I'll need help.
>
> Looks like in Jira we currently have:
>  26 MR blockers: http://tinyurl.com/23s5vzg
>  17 HDFS blockers: http://tinyurl.com/27w82vl
>  6 HADOOP blockers: http://tinyurl.com/2bmycl2
> Now that we are past feature freeze (i.e. branch creation), not all of
> these are truly blockers.  I'll start cleaning up the list and ping folks
> after the holidays.
>
> Cheers,
> Nige
>
> On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
> >
> > On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
> >
> >>> Everyone who has discussed this patch has said it isn't critical to
> hadoop, and It's holding up everything else 0.22 is going to bring.
> >>
> >> I disagree that it isn't critical to Hadoop, but I'm not holding up
> 0.22. I'm just not volunteering to spend my time working on it, if it
> doesn't have the features that I think it needs.
> >>
> >> -- Owen
> >>
> >
> > That's a fair point.. and this is a volunteer effort.
> > Do we have anybody else who is willing to be the release manager for 22
> with 6685?
> >
>
>