I've created a release candidate for hadoop-0.23.2 that I would like to release.

It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

What are the issues fixed / features added in 0.23.2 compared to 0.23.1 ?

Thanks

On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> I've created a release candidate for hadoop-0.23.2 that I would like to
> release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>
> The maven artifacts are available via repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't have NN HA etc.

As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a couple weeks with NN HA, PB for HDFS etc.

thanks,
Arun

On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:

> What are the issues fixed / features added in 0.23.2 compared to 0.23.1 ?
>
> Thanks
>
> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>
>> I've created a release candidate for hadoop-0.23.2 that I would like to
>> release.
>>
>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>>
>> The maven artifacts are available via repository.apache.org.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>> thanks,
>> Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Hey Arun,

This vote passed a week or so ago, let's make it official?

Also, are you still planning to roll a hadoop-2.0.0-rc0 of branch-2
this week?  I think we should do that soon, if you're not planning to
do this holler and I'd be happy to.  There's only 1 blocker left
(http://bit.ly/I55LAd) and it's patch available, I think we should
role an rc from branch-2 when it's merged.

Thanks,
Eli

On Thu, Mar 29, 2012 at 4:07 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> 0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't have NN HA etc.
>
> As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a couple weeks with NN HA, PB for HDFS etc.
>
> thanks,
> Arun
>
> On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:
>
>> What are the issues fixed / features added in 0.23.2 compared to 0.23.1 ?
>>
>> Thanks
>>
>> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>>
>>> I've created a release candidate for hadoop-0.23.2 that I would like to
>>> release.
>>>
>>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>>>
>>> The maven artifacts are available via repository.apache.org.
>>>
>>> Please try the release and vote; the vote will run for the usual 7 days.
>>>
>>> thanks,
>>> Arun
>>>
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
>>>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

Sorry, I missed sending the note - I'm rolling rc1 since rc0 didn't have a security fix for LCE.

Arun

On Apr 19, 2012, at 12:24 AM, Eli Collins wrote:

> Hey Arun,
>
> This vote passed a week or so ago, let's make it official?
>
> Also, are you still planning to roll a hadoop-2.0.0-rc0 of branch-2
> this week?  I think we should do that soon, if you're not planning to
> do this holler and I'd be happy to.  There's only 1 blocker left
> (http://bit.ly/I55LAd) and it's patch available, I think we should
> role an rc from branch-2 when it's merged.
>
> Thanks,
> Eli
>
> On Thu, Mar 29, 2012 at 4:07 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>> 0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't have NN HA etc.
>>
>> As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a couple weeks with NN HA, PB for HDFS etc.
>>
>> thanks,
>> Arun
>>
>> On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:
>>
>>> What are the issues fixed / features added in 0.23.2 compared to 0.23.1 ?
>>>
>>> Thanks
>>>
>>> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>>>
>>>> I've created a release candidate for hadoop-0.23.2 that I would like to
>>>> release.
>>>>
>>>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>>>>
>>>> The maven artifacts are available via repository.apache.org.
>>>>
>>>> Please try the release and vote; the vote will run for the usual 7 days.
>>>>
>>>> thanks,
>>>> Arun
>>>>
>>>> --
>>>> Arun C. Murthy
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>>
>>>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Yep, makes sense - I'll roll an rc0 for 2.0 after.

However, we should consider whether HDFS protocols are 'ready' for us to commit to them for the foreseeable future, my sense is that it's a tad early - particularly with auto-failover not complete.

Thus, we have a couple of options:
a) Call the first release here as *2.0.0-alpha* version (lots of ASF projects do this).
b) Just go with 2.0.0 and deem 2.0.x or 2.1.x as the first stable release and fwd-compatible release later.

Given this is a major release (unlike something obscure like hadoop-0.23.0) I'm inclined to go with a) i.e. hadoop-2.0.0-alpha.

Thoughts?

Arun

On Apr 19, 2012, at 12:24 AM, Eli Collins wrote:

> Hey Arun,
>
> This vote passed a week or so ago, let's make it official?
>
> Also, are you still planning to roll a hadoop-2.0.0-rc0 of branch-2
> this week?  I think we should do that soon, if you're not planning to
> do this holler and I'd be happy to.  There's only 1 blocker left
> (http://bit.ly/I55LAd) and it's patch available, I think we should
> role an rc from branch-2 when it's merged.
>
> Thanks,
> Eli
>
> On Thu, Mar 29, 2012 at 4:07 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>> 0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't have NN HA etc.
>>
>> As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a couple weeks with NN HA, PB for HDFS etc.
>>
>> thanks,
>> Arun
>>
>> On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:
>>
>>> What are the issues fixed / features added in 0.23.2 compared to 0.23.1 ?
>>>
>>> Thanks
>>>
>>> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>>>
>>>> I've created a release candidate for hadoop-0.23.2 that I would like to
>>>> release.
>>>>
>>>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>>>>
>>>> The maven artifacts are available via repository.apache.org.
>>>>
>>>> Please try the release and vote; the vote will run for the usual 7 days.
>>>>
>>>> thanks,
>>>> Arun
>>>>
>>>> --
>>>> Arun C. Murthy
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>>
>>>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

On Thu, Apr 19, 2012 at 11:45 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> Yep, makes sense - I'll roll an rc0 for 2.0 after.
>
> However, we should consider whether HDFS protocols are 'ready' for us to commit to them for the foreseeable future, my sense is that it's a tad early - particularly with auto-failover not complete.
>
> Thus, we have a couple of options:
> a) Call the first release here as *2.0.0-alpha* version (lots of ASF projects do this).
> b) Just go with 2.0.0 and deem 2.0.x or 2.1.x as the first stable release and fwd-compatible release later.
>
> Given this is a major release (unlike something obscure like hadoop-0.23.0) I'm inclined to go with a) i.e. hadoop-2.0.0-alpha.
>
> Thoughts?
>

Agree that we're a little too early on the HDFS protocol side, think
MR2 is probably in a similar boat wrt stability as well.

+1 to option a, calling it hadoop-2.0.0-alpha seems most appropriate.

Thanks Arun.
> Arun
>
> On Apr 19, 2012, at 12:24 AM, Eli Collins wrote:
>
>> Hey Arun,
>>
>> This vote passed a week or so ago, let's make it official?
>>
>> Also, are you still planning to roll a hadoop-2.0.0-rc0 of branch-2
>> this week?  I think we should do that soon, if you're not planning to
>> do this holler and I'd be happy to.  There's only 1 blocker left
>> (http://bit.ly/I55LAd) and it's patch available, I think we should
>> role an rc from branch-2 when it's merged.
>>
>> Thanks,
>> Eli
>>
>> On Thu, Mar 29, 2012 at 4:07 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>>> 0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't have NN HA etc.
>>>
>>> As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a couple weeks with NN HA, PB for HDFS etc.
>>>
>>> thanks,
>>> Arun
>>>
>>> On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:
>>>
>>>> What are the issues fixed / features added in 0.23.2 compared to 0.23.1 ?
>>>>
>>>> Thanks
>>>>
>>>> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> I've created a release candidate for hadoop-0.23.2 that I would like to
>>>>> release.
>>>>>
>>>>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>>>>>
>>>>> The maven artifacts are available via repository.apache.org.
>>>>>
>>>>> Please try the release and vote; the vote will run for the usual 7 days.
>>>>>
>>>>> thanks,
>>>>> Arun
>>>>>
>>>>> --
>>>>> Arun C. Murthy
>>>>> Hortonworks Inc.
>>>>> http://hortonworks.com/
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

On Thu, Apr 19, 2012 at 12:26 PM, Eli Collins <[EMAIL PROTECTED]> wrote:

> On Thu, Apr 19, 2012 at 11:45 AM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> > Yep, makes sense - I'll roll an rc0 for 2.0 after.
> >
> > However, we should consider whether HDFS protocols are 'ready' for us to
> commit to them for the foreseeable future, my sense is that it's a tad
> early - particularly with auto-failover not complete.
> >
> > Thus, we have a couple of options:
> > a) Call the first release here as *2.0.0-alpha* version (lots of ASF
> projects do this).
> > b) Just go with 2.0.0 and deem 2.0.x or 2.1.x as the first stable
> release and fwd-compatible release later.
> >
> > Given this is a major release (unlike something obscure like
> hadoop-0.23.0) I'm inclined to go with a) i.e. hadoop-2.0.0-alpha.
> >
> > Thoughts?
> >
>
> Agree that we're a little too early on the HDFS protocol side, think
> MR2 is probably in a similar boat wrt stability as well.
>
> +1 to option a, calling it hadoop-2.0.0-alpha seems most appropriate.
>

Regarding protocols:
+1 to _not_ locking down "cluster-internal" wire compatibility at this
point. i.e we can break DN<->NN, or NN<->SBN, or Admin command -> NN
compatibility still.
+1 to locking down client wire compatibility with the release of 2.0. After
2.0 is released I would like to see all 2.0.x clients continue to be
compatible. Now that we are protobuf-ified, I think this is doable.
Should we open a separate discussion thread for the above?

Regarding version numbering: either of the proposals seems fine by me.

-Todd

 > Arun
> >
> > On Apr 19, 2012, at 12:24 AM, Eli Collins wrote:
> >
> >> Hey Arun,
> >>
> >> This vote passed a week or so ago, let's make it official?
> >>
> >> Also, are you still planning to roll a hadoop-2.0.0-rc0 of branch-2
> >> this week?  I think we should do that soon, if you're not planning to
> >> do this holler and I'd be happy to.  There's only 1 blocker left
> >> (http://bit.ly/I55LAd) and it's patch available, I think we should
> >> role an rc from branch-2 when it's merged.
> >>
> >> Thanks,
> >> Eli
> >>
> >> On Thu, Mar 29, 2012 at 4:07 PM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >>> 0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't
> have NN HA etc.
> >>>
> >>> As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a
> couple weeks with NN HA, PB for HDFS etc.
> >>>
> >>> thanks,
> >>> Arun
> >>>
> >>> On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:
> >>>
> >>>> What are the issues fixed / features added in 0.23.2 compared to
> 0.23.1 ?
> >>>>
> >>>> Thanks
> >>>>
> >>>> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >>>>
> >>>>> I've created a release candidate for hadoop-0.23.2 that I would like
> to
> >>>>> release.
> >>>>>
> >>>>> It is available at:
> http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
> >>>>>
> >>>>> The maven artifacts are available via repository.apache.org.
> >>>>>
> >>>>> Please try the release and vote; the vote will run for the usual 7
> days.
> >>>>>
> >>>>> thanks,
> >>>>> Arun
> >>>>>
> >>>>> --
> >>>>> Arun C. Murthy
> >>>>> Hortonworks Inc.
> >>>>> http://hortonworks.com/
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>> --
> >>> Arun C. Murthy
> >>> Hortonworks Inc.
> >>> http://hortonworks.com/
> >>>
> >>>
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>

--
Todd Lipcon
Software Engineer, Cloudera

-1 for option b. KDE did that with 4.0, and it caused a lot of confusion,
frustration and occasional mocking, although it was very well known and
documented that 4.0 series were still beta quality.

+1 for option a. I would image we can go with 2.0.x-alpha releases, and
when we are ready to go stable, call it 2.1 release, wdyt?

Enis

On Thu, Apr 19, 2012 at 12:54 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Thu, Apr 19, 2012 at 12:26 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>
> > On Thu, Apr 19, 2012 at 11:45 AM, Arun C Murthy <[EMAIL PROTECTED]>
> > wrote:
> > > Yep, makes sense - I'll roll an rc0 for 2.0 after.
> > >
> > > However, we should consider whether HDFS protocols are 'ready' for us
> to
> > commit to them for the foreseeable future, my sense is that it's a tad
> > early - particularly with auto-failover not complete.
> > >
> > > Thus, we have a couple of options:
> > > a) Call the first release here as *2.0.0-alpha* version (lots of ASF
> > projects do this).
> > > b) Just go with 2.0.0 and deem 2.0.x or 2.1.x as the first stable
> > release and fwd-compatible release later.
> > >
> > > Given this is a major release (unlike something obscure like
> > hadoop-0.23.0) I'm inclined to go with a) i.e. hadoop-2.0.0-alpha.
> > >
> > > Thoughts?
> > >
> >
> > Agree that we're a little too early on the HDFS protocol side, think
> > MR2 is probably in a similar boat wrt stability as well.
> >
> > +1 to option a, calling it hadoop-2.0.0-alpha seems most appropriate.
> >
>
> Regarding protocols:
> +1 to _not_ locking down "cluster-internal" wire compatibility at this
> point. i.e we can break DN<->NN, or NN<->SBN, or Admin command -> NN
> compatibility still.
> +1 to locking down client wire compatibility with the release of 2.0. After
> 2.0 is released I would like to see all 2.0.x clients continue to be
> compatible. Now that we are protobuf-ified, I think this is doable.
> Should we open a separate discussion thread for the above?
>
> Regarding version numbering: either of the proposals seems fine by me.
>
> -Todd
>
>  > Arun
> > >
> > > On Apr 19, 2012, at 12:24 AM, Eli Collins wrote:
> > >
> > >> Hey Arun,
> > >>
> > >> This vote passed a week or so ago, let's make it official?
> > >>
> > >> Also, are you still planning to roll a hadoop-2.0.0-rc0 of branch-2
> > >> this week?  I think we should do that soon, if you're not planning to
> > >> do this holler and I'd be happy to.  There's only 1 blocker left
> > >> (http://bit.ly/I55LAd) and it's patch available, I think we should
> > >> role an rc from branch-2 when it's merged.
> > >>
> > >> Thanks,
> > >> Eli
> > >>
> > >> On Thu, Mar 29, 2012 at 4:07 PM, Arun C Murthy <[EMAIL PROTECTED]>
> > wrote:
> > >>> 0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't
> > have NN HA etc.
> > >>>
> > >>> As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a
> > couple weeks with NN HA, PB for HDFS etc.
> > >>>
> > >>> thanks,
> > >>> Arun
> > >>>
> > >>> On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:
> > >>>
> > >>>> What are the issues fixed / features added in 0.23.2 compared to
> > 0.23.1 ?
> > >>>>
> > >>>> Thanks
> > >>>>
> > >>>> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]
> >
> > wrote:
> > >>>>
> > >>>>> I've created a release candidate for hadoop-0.23.2 that I would
> like
> > to
> > >>>>> release.
> > >>>>>
> > >>>>> It is available at:
> > http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
> > >>>>>
> > >>>>> The maven artifacts are available via repository.apache.org.
> > >>>>>
> > >>>>> Please try the release and vote; the vote will run for the usual 7
> > days.
> > >>>>>
> > >>>>> thanks,
> > >>>>> Arun
> > >>>>>
> > >>>>> --
> > >>>>> Arun C. Murthy
> > >>>>> Hortonworks Inc.
> > >>>>> http://hortonworks.com/
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>> --
> > >>> Arun C. Murthy
> > >>> Hortonworks Inc.
> > >>> http://hortonworks.com/
> > >>>
> > >>>
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.

I believe it's premature to release a non-alpha. branch-2.0 does not contain a full working implementation of host-based tokens that was introduced in 1.x (yes, it was done out of order...).  This is a very important feature that prevents tokens from being invalidated when a host's ip changes.  The token implementation in 2.0 requires daemons to be restarted and/or jobs resubmitted when the ip of an NN is changed (ex. an upgrade).  Host-based tokens prevents the need to restart all cluster that access a remote cluster that is upgraded.

I've been actively working on the yarn side and close to completion.  Until that's complete, I feel we should consider 2.x an alpha so there's not an omission of a major feature.

Daryn
On Apr 19, 2012, at 1:45 PM, Arun C Murthy wrote:

> Yep, makes sense - I'll roll an rc0 for 2.0 after.
>
> However, we should consider whether HDFS protocols are 'ready' for us to commit to them for the foreseeable future, my sense is that it's a tad early - particularly with auto-failover not complete.
>
> Thus, we have a couple of options:
> a) Call the first release here as *2.0.0-alpha* version (lots of ASF projects do this).
> b) Just go with 2.0.0 and deem 2.0.x or 2.1.x as the first stable release and fwd-compatible release later.
>
> Given this is a major release (unlike something obscure like hadoop-0.23.0) I'm inclined to go with a) i.e. hadoop-2.0.0-alpha.
>
> Thoughts?
>
> Arun
>
> On Apr 19, 2012, at 12:24 AM, Eli Collins wrote:
>
>> Hey Arun,
>>
>> This vote passed a week or so ago, let's make it official?
>>
>> Also, are you still planning to roll a hadoop-2.0.0-rc0 of branch-2
>> this week?  I think we should do that soon, if you're not planning to
>> do this holler and I'd be happy to.  There's only 1 blocker left
>> (http://bit.ly/I55LAd) and it's patch available, I think we should
>> role an rc from branch-2 when it's merged.
>>
>> Thanks,
>> Eli
>>
>> On Thu, Mar 29, 2012 at 4:07 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>>> 0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't have NN HA etc.
>>>
>>> As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a couple weeks with NN HA, PB for HDFS etc.
>>>
>>> thanks,
>>> Arun
>>>
>>> On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:
>>>
>>>> What are the issues fixed / features added in 0.23.2 compared to 0.23.1 ?
>>>>
>>>> Thanks
>>>>
>>>> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> I've created a release candidate for hadoop-0.23.2 that I would like to
>>>>> release.
>>>>>
>>>>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>>>>>
>>>>> The maven artifacts are available via repository.apache.org.
>>>>>
>>>>> Please try the release and vote; the vote will run for the usual 7 days.
>>>>>
>>>>> thanks,
>>>>> Arun
>>>>>
>>>>> --
>>>>> Arun C. Murthy
>>>>> Hortonworks Inc.
>>>>> http://hortonworks.com/
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

On Fri, Apr 20, 2012 at 7:28 AM, Daryn Sharp <[EMAIL PROTECTED]> wrote:

> I believe it's premature to release a non-alpha. branch-2.0 does not
> contain a full working implementation of host-based tokens that was
> introduced in 1.x (yes, it was done out of order...).  This is a very
> important feature that prevents tokens from being invalidated when a host's
> ip changes.  The token implementation in 2.0 requires daemons to be
> restarted and/or jobs resubmitted when the ip of an NN is changed (ex. an
> upgrade).  Host-based tokens prevents the need to restart all cluster that
> access a remote cluster that is upgraded.
>

I think blocking the release of a non-alpha on this feature is a bit nutty.
Not to undermine the work, but it only affects users who run security and
want to be able to move a NN/JT from one IP to another without killing
currently running jobs. Only a small fraction of the user base enables
security at all, and an even smaller fraction regularly wants to migrate a
master to a new IP mid-job.
>
> I've been actively working on the yarn side and close to completion.
>  Until that's complete, I feel we should consider 2.x an alpha so there's
> not an omission of a major feature.
>

I'd call it a minor feature.

-Todd
>
> On Apr 19, 2012, at 1:45 PM, Arun C Murthy wrote:
>
> > Yep, makes sense - I'll roll an rc0 for 2.0 after.
> >
> > However, we should consider whether HDFS protocols are 'ready' for us to
> commit to them for the foreseeable future, my sense is that it's a tad
> early - particularly with auto-failover not complete.
> >
> > Thus, we have a couple of options:
> > a) Call the first release here as *2.0.0-alpha* version (lots of ASF
> projects do this).
> > b) Just go with 2.0.0 and deem 2.0.x or 2.1.x as the first stable
> release and fwd-compatible release later.
> >
> > Given this is a major release (unlike something obscure like
> hadoop-0.23.0) I'm inclined to go with a) i.e. hadoop-2.0.0-alpha.
> >
> > Thoughts?
> >
> > Arun
> >
> > On Apr 19, 2012, at 12:24 AM, Eli Collins wrote:
> >
> >> Hey Arun,
> >>
> >> This vote passed a week or so ago, let's make it official?
> >>
> >> Also, are you still planning to roll a hadoop-2.0.0-rc0 of branch-2
> >> this week?  I think we should do that soon, if you're not planning to
> >> do this holler and I'd be happy to.  There's only 1 blocker left
> >> (http://bit.ly/I55LAd) and it's patch available, I think we should
> >> role an rc from branch-2 when it's merged.
> >>
> >> Thanks,
> >> Eli
> >>
> >> On Thu, Mar 29, 2012 at 4:07 PM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >>> 0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't
> have NN HA etc.
> >>>
> >>> As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a
> couple weeks with NN HA, PB for HDFS etc.
> >>>
> >>> thanks,
> >>> Arun
> >>>
> >>> On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:
> >>>
> >>>> What are the issues fixed / features added in 0.23.2 compared to
> 0.23.1 ?
> >>>>
> >>>> Thanks
> >>>>
> >>>> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >>>>
> >>>>> I've created a release candidate for hadoop-0.23.2 that I would like
> to
> >>>>> release.
> >>>>>
> >>>>> It is available at:
> http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
> >>>>>
> >>>>> The maven artifacts are available via repository.apache.org.
> >>>>>
> >>>>> Please try the release and vote; the vote will run for the usual 7
> days.
> >>>>>
> >>>>> thanks,
> >>>>> Arun
> >>>>>
> >>>>> --
> >>>>> Arun C. Murthy
> >>>>> Hortonworks Inc.
> >>>>> http://hortonworks.com/
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>> --
> >>> Arun C. Murthy
> >>> Hortonworks Inc.
> >>> http://hortonworks.com/
> >>>
> >>>
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
>
--
Todd Lipcon
Software Engineer, Cloudera

Good discussion.  +1 to shipping it as 2.0-alpha

I think there are a number of areas, some enumerated already where the 2 line is immature and is likely to change.  I think we should enumerate the areas we expect change and determine which we are comfortable changing in later dot releases and which we need before we can declare the 2 line to be stable.

We should also make sure we are all on the same page in terms of rules of the road for forwards and backwards compatibility between various releases on the 2 line.

Questions:

- Do 2.0 clients work on all future 2 line releases without recompilation?

- Do we know how 1.0 clients migrate to 2.0?  Are there changes we can make to support that?

- What major features are coming soon that will require protocol change
-- Service directory / availability -- Need to be able to change the addresses of services

- Are the YARN APIs complete enough?  I'd like to see a couple of new frameworks implemented to work out the APIs
-- MPI - A classic with new needs
-- Maybe supporting HBase as a service within YARN?  This will test APIs
-- Headless Map-Reduce or Pig jobs?  One should be able to run them without a fat client...
-- Job history tracking for arbitrary frameworks without new server code / framework?

----

What do folks think of this list?  What would you add?

On Apr 20, 2012, at 11:41 AM, Todd Lipcon wrote:

> On Fri, Apr 20, 2012 at 7:28 AM, Daryn Sharp <[EMAIL PROTECTED]> wrote:
>
>> I believe it's premature to release a non-alpha. branch-2.0 does not
>> contain a full working implementation of host-based tokens that was
>> introduced in 1.x (yes, it was done out of order...).  This is a very
>> important feature that prevents tokens from being invalidated when a host's
>> ip changes.  The token implementation in 2.0 requires daemons to be
>> restarted and/or jobs resubmitted when the ip of an NN is changed (ex. an
>> upgrade).  Host-based tokens prevents the need to restart all cluster that
>> access a remote cluster that is upgraded.
>>
>
> I think blocking the release of a non-alpha on this feature is a bit nutty.
> Not to undermine the work, but it only affects users who run security and
> want to be able to move a NN/JT from one IP to another without killing
> currently running jobs. Only a small fraction of the user base enables
> security at all, and an even smaller fraction regularly wants to migrate a
> master to a new IP mid-job.
>
>
>>
>> I've been actively working on the yarn side and close to completion.
>> Until that's complete, I feel we should consider 2.x an alpha so there's
>> not an omission of a major feature.
>>
>
> I'd call it a minor feature.
>
> -Todd
>
>
>>
>> On Apr 19, 2012, at 1:45 PM, Arun C Murthy wrote:
>>
>>> Yep, makes sense - I'll roll an rc0 for 2.0 after.
>>>
>>> However, we should consider whether HDFS protocols are 'ready' for us to
>> commit to them for the foreseeable future, my sense is that it's a tad
>> early - particularly with auto-failover not complete.
>>>
>>> Thus, we have a couple of options:
>>> a) Call the first release here as *2.0.0-alpha* version (lots of ASF
>> projects do this).
>>> b) Just go with 2.0.0 and deem 2.0.x or 2.1.x as the first stable
>> release and fwd-compatible release later.
>>>
>>> Given this is a major release (unlike something obscure like
>> hadoop-0.23.0) I'm inclined to go with a) i.e. hadoop-2.0.0-alpha.
>>>
>>> Thoughts?
>>>
>>> Arun
>>>
>>> On Apr 19, 2012, at 12:24 AM, Eli Collins wrote:
>>>
>>>> Hey Arun,
>>>>
>>>> This vote passed a week or so ago, let's make it official?
>>>>
>>>> Also, are you still planning to roll a hadoop-2.0.0-rc0 of branch-2
>>>> this week?  I think we should do that soon, if you're not planning to
>>>> do this holler and I'd be happy to.  There's only 1 blocker left
>>>> (http://bit.ly/I55LAd) and it's patch available, I think we should
>>>> role an rc from branch-2 when it's merged.
>>>>
>>>> Thanks,
>>>> Eli
>>>>
>>>> On Thu, Mar 29, 2012 at 4:07 PM, Arun C Murthy <[EMAIL PROTECTED]>

On 23 April 2012 22:54, Eric Baldeschwieler <[EMAIL PROTECTED]> wrote:

>
> - Are the YARN APIs complete enough?  I'd like to see a couple of new
> frameworks implemented to work out the APIs
> -- MPI - A classic with new needs
> -- Maybe supporting HBase as a service within YARN?  This will test APIs
> -- Headless Map-Reduce or Pig jobs?  One should be able to run them
> without a fat client...
> -- Job history tracking for arbitrary frameworks without new server code /
> framework?
>
>
-Graphing: Hama & Giraph. these would be ideal as they aren't at 1.x
release stage yet, and because it makes so much sense to run them within
the Hadoop cluster where the data lives. I think Giraph already sneaks in
an an MR job, because the limit on Job Counters in the 1.x branch is
limiting the number of supersteps.

-I've ported the remote shell app to Groovy, though not actually tested it
yet. What I'd like is to deploy something on every node that can take a
list of nodes and a start time, and trigger an attempt to send data to the
destination -that way you can qualify a cluster's bandwidth. for that I'd
really like a way for any user w/ admin rights to deploy something on a
named machine even if there are no free resources.

Hi Steve and Eric,

sorry to spam your vote thread.
Hama is currently in the process of releasing 0.5.0-incubating including
Hadoop 1.0 and YARN module with 0.23.1.

Let us know if we can check something with 0.23.2. Just post a comment [1].

Thanks and bye.

[1] https://issues.apache.org/jira/browse/HAMA-431

Am 24. April 2012 12:47 schrieb Steve Loughran <[EMAIL PROTECTED]>:

> On 23 April 2012 22:54, Eric Baldeschwieler <[EMAIL PROTECTED]>
> wrote:
>
> >
> > - Are the YARN APIs complete enough?  I'd like to see a couple of new
> > frameworks implemented to work out the APIs
> > -- MPI - A classic with new needs
> > -- Maybe supporting HBase as a service within YARN?  This will test APIs
> > -- Headless Map-Reduce or Pig jobs?  One should be able to run them
> > without a fat client...
> > -- Job history tracking for arbitrary frameworks without new server code
> /
> > framework?
> >
> >
> -Graphing: Hama & Giraph. these would be ideal as they aren't at 1.x
> release stage yet, and because it makes so much sense to run them within
> the Hadoop cluster where the data lives. I think Giraph already sneaks in
> an an MR job, because the limit on Job Counters in the 1.x branch is
> limiting the number of supersteps.
>
> -I've ported the remote shell app to Groovy, though not actually tested it
> yet. What I'd like is to deploy something on every node that can take a
> list of nodes and a start time, and trigger an attempt to send data to the
> destination -that way you can qualify a cluster's bandwidth. for that I'd
> really like a way for any user w/ admin rights to deploy something on a
> named machine even if there are no free resources.
>

--
Thomas Jungblut
Berlin <[EMAIL PROTECTED]>

+1

I downloaded the RC.  Ran several different jobs on a one node cluster without security.  Killed a few processes while the job was running and watched it recover.

--Bobby Evans
On 3/29/12 6:07 PM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:

0.23.2 is just  a small set of bug-fixes on top of 0.23.1 and doesn't have NN HA etc.

As I've noted separately, I plan to put out a hadoop-2.0.0-rc0 in a couple weeks with NN HA, PB for HDFS etc.

thanks,
Arun

On Mar 29, 2012, at 3:55 PM, Ted Yu wrote:

> What are the issues fixed / features added in 0.23.2 compared to 0.23.1 ?
>
> Thanks
>
> On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>
>> I've created a release candidate for hadoop-0.23.2 that I would like to
>> release.
>>
>> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>>
>> The maven artifacts are available via repository.apache.org.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>> thanks,
>> Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

+1.
Verified the sums.
Built from the source tgz and deployed a single node cluster. Ran a couple
of test jobs which ran as they should.

- Sid

On Thu, Mar 29, 2012 at 3:45 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> I've created a release candidate for hadoop-0.23.2 that I would like to
> release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>
> The maven artifacts are available via repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

+1, I downloaded and checked this rc, some of the things I checked:

  - html documentation present in binary: share/doc/hadoop/index.html
  - release is present at:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoo
p-common/0.23.2/
  - LICENSE, NOTICE and REAEDME are good
  - md5 and signatures good
  - src tarball builds

Testing: All testing on 2 nodes, unsecured
  - all daemons started - hdfs/yarn/history server
  - Ran wordcount, sleep and both passed

Thanks,
Tom Graves

On 3/29/12 5:45 PM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:

> I've created a release candidate for hadoop-0.23.2 that I would like to
> release.
>
> It is available at: http://people.apache.org/~acmurthy/hadoop-0.23.2-rc0/
>
> The maven artifacts are available via repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>