The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
the 0.20 release series when the "new" (Context Objects) MapReduce API
was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
was not complete in 0.20 and most users stayed with the old API. This
has led to the confusing situation where the old API is generally
recommended, even though it is deprecated.

To remedy this situation I suggest that we remove deprecations from
the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
MAPREDUCE-1623 for the latter). This would mean a few things:

* The next 0.20 release would have a non-deprecated old API.
* The forthcoming 0.21 release would have a "Stable" (non-deprecated)
old API, and a "Evolving" new API.
* For some pre-1.0 release (perhaps 0.22), the old API could be
deprecated again, and the new API marked as "Stable".
* In the 1.0 release it would be possible to remove the old API.

Thoughts?

Tom

+1. Currently there is almost no way to write a moderately complex MR
job that doesn't spew deprecation warnings. It serves to endlessly
confuse newcomers to Hadoop.

On Wed, Apr 21, 2010 at 5:24 PM, Tom White <[EMAIL PROTECTED]> wrote:
> The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
> the 0.20 release series when the "new" (Context Objects) MapReduce API
> was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
> was not complete in 0.20 and most users stayed with the old API. This
> has led to the confusing situation where the old API is generally
> recommended, even though it is deprecated.
>
> To remedy this situation I suggest that we remove deprecations from
> the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
> MAPREDUCE-1623 for the latter). This would mean a few things:
>
> * The next 0.20 release would have a non-deprecated old API.
> * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
> old API, and a "Evolving" new API.
> * For some pre-1.0 release (perhaps 0.22), the old API could be
> deprecated again, and the new API marked as "Stable".
> * In the 1.0 release it would be possible to remove the old API.
>
> Thoughts?
>
> Tom
>

--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

On the various pieces, I think:

0.20: -0 for removing the deprecation, +1 for improving the
deprecation message with links to the corresponding class.

0.21: new core api should be stable except for Job and Cluster
        new library code should be evolving
        -1 for removing the deprecation, we need to

0.22: all of the new api should be stable and the old api deprecated.

> Currently there is almost no way to write a moderately complex MR job that doesn't spew deprecation warnings.

That is false in 0.21.

-- Owen

+1

On Wed, Apr 21, 2010 at 2:24 PM, Tom White <[EMAIL PROTECTED]> wrote:
> The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
> the 0.20 release series when the "new" (Context Objects) MapReduce API
> was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
> was not complete in 0.20 and most users stayed with the old API. This
> has led to the confusing situation where the old API is generally
> recommended, even though it is deprecated.
>
> To remedy this situation I suggest that we remove deprecations from
> the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
> MAPREDUCE-1623 for the latter). This would mean a few things:
>
> * The next 0.20 release would have a non-deprecated old API.
> * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
> old API, and a "Evolving" new API.
> * For some pre-1.0 release (perhaps 0.22), the old API could be
> deprecated again, and the new API marked as "Stable".
> * In the 1.0 release it would be possible to remove the old API.
>
> Thoughts?
>
> Tom
>

Owen O'Malley wrote:
> 0.21: new core api should be stable except for Job and Cluster
>         new library code should be evolving
>         -1 for removing the deprecation, we need to

The Job API is central to the new API, no?  Should we encourage
applications to move to the new API if it's not entirely stable?  Can
you provide a more detailed rationale for deprecating the only
fully-stable API?

Thanks,

Doug

+1 for undeprecating. I think it's  just being pragmatic.

Arun

On Apr 21, 2010, at 2:26 PM, "Tom White" <[EMAIL PROTECTED]> wrote:

> The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
> the 0.20 release series when the "new" (Context Objects) MapReduce API
> was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
> was not complete in 0.20 and most users stayed with the old API. This
> has led to the confusing situation where the old API is generally
> recommended, even though it is deprecated.
>
> To remedy this situation I suggest that we remove deprecations from
> the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
> MAPREDUCE-1623 for the latter). This would mean a few things:
>
> * The next 0.20 release would have a non-deprecated old API.
> * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
> old API, and a "Evolving" new API.
> * For some pre-1.0 release (perhaps 0.22), the old API could be
> deprecated again, and the new API marked as "Stable".
> * In the 1.0 release it would be possible to remove the old API.
>
> Thoughts?
>
> Tom

+1 for un-deprecating. I'm just being pragmatic.

Arun

On Apr 21, 2010, at 2:24 PM, Tom White wrote:

> The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
> the 0.20 release series when the "new" (Context Objects) MapReduce API
> was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
> was not complete in 0.20 and most users stayed with the old API. This
> has led to the confusing situation where the old API is generally
> recommended, even though it is deprecated.
>
> To remedy this situation I suggest that we remove deprecations from
> the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
> MAPREDUCE-1623 for the latter). This would mean a few things:
>
> * The next 0.20 release would have a non-deprecated old API.
> * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
> old API, and a "Evolving" new API.
> * For some pre-1.0 release (perhaps 0.22), the old API could be
> deprecated again, and the new API marked as "Stable".
> * In the 1.0 release it would be possible to remove the old API.
>
> Thoughts?
>
> Tom

+1 (mainly to avoid confusing new comers as Eric Sammer correctly
indicated).

On 4/22/2010 9:15 AM, Doug Cutting wrote:
> Owen O'Malley wrote:
>> 0.21: new core api should be stable except for Job and Cluster
>>         new library code should be evolving
>>         -1 for removing the deprecation, we need to
>
> The Job API is central to the new API, no?  Should we encourage
> applications to move to the new API if it's not entirely stable?  Can
> you provide a more detailed rationale for deprecating the only
> fully-stable API?
>
> Thanks,
>
> Doug

Speaking for one power user (Pig) that did move to the new APIs,  
moving that interface to evolving is a little unsettling.  Is there a  
feel for how much the new API is going to change?

Alan.

On Apr 21, 2010, at 2:24 PM, Tom White wrote:

> The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
> the 0.20 release series when the "new" (Context Objects) MapReduce API
> was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
> was not complete in 0.20 and most users stayed with the old API. This
> has led to the confusing situation where the old API is generally
> recommended, even though it is deprecated.
>
> To remedy this situation I suggest that we remove deprecations from
> the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
> MAPREDUCE-1623 for the latter). This would mean a few things:
>
> * The next 0.20 release would have a non-deprecated old API.
> * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
> old API, and a "Evolving" new API.
> * For some pre-1.0 release (perhaps 0.22), the old API could be
> deprecated again, and the new API marked as "Stable".
> * In the 1.0 release it would be possible to remove the old API.
>
> Thoughts?
>
> Tom

> * The next 0.20 release would have a non-deprecated old API.

+1

> * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
> old API, and a "Evolving" new API.

+1

> * For some pre-1.0 release (perhaps 0.22), the old API could be
> deprecated again, and the new API marked as "Stable".
> * In the 1.0 release it would be possible to remove the old API.
thinking this is a different discussion.

but my preference is that 0.20 become 1.0

buying time to evolve the evolving apis and build/dependency system over the next year(s) into a 2.0

this could bite me as I do like some of the changes i had to effect in my cascading wip 2.0 api to support the 'new' apis.

but a 1.0 will send a very useful message to potential users

chris

--
Chris K Wensel
[EMAIL PROTECTED]
http://www.concurrentinc.com

+1 for removing deprecation in 0.20.

+0 for removing deprecation in 0.21

Thanks
Amareshwari
On 4/22/10 7:25 PM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote:

On the various pieces, I think:

0.20: -0 for removing the deprecation, +1 for improving the
deprecation message with links to the corresponding class.

0.21: new core api should be stable except for Job and Cluster
        new library code should be evolving
        -1 for removing the deprecation, we need to

0.22: all of the new api should be stable and the old api deprecated.

> Currently there is almost no way to write a moderately complex MR job that doesn't spew deprecation warnings.

That is false in 0.21.

-- Owen

Alan,

On Apr 22, 2010, at 12:12 PM, Alan Gates wrote:

> Speaking for one power user (Pig) that did move to the new APIs,  
> moving that interface to evolving is a little unsettling.  Is there  
> a feel for how much the new API is going to change?
>

  The intent isn't to mark the 'new' apis as 'Evolving' to change them  
willy-nilly... please don't read it so!

  This is just a pragmatic proposal to reflect that the 'old' apis  
will, for lack of stabilization of new apis, be supported.

  Given that, the new apis could mostly be stable, but for Job and  
Cluster - is that reasonable? This will ensure we send the right  
message all concerned regarding stability of o.a.h.mapreduce.{Mapper|
Reducer|...}. Thoughts?

Arun

> Alan.
>
> On Apr 21, 2010, at 2:24 PM, Tom White wrote:
>
>> The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
>> the 0.20 release series when the "new" (Context Objects) MapReduce  
>> API
>> was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
>> was not complete in 0.20 and most users stayed with the old API. This
>> has led to the confusing situation where the old API is generally
>> recommended, even though it is deprecated.
>>
>> To remedy this situation I suggest that we remove deprecations from
>> the old API in 0.20 and trunk, and mark the new API as  
>> "Evolving" (see
>> MAPREDUCE-1623 for the latter). This would mean a few things:
>>
>> * The next 0.20 release would have a non-deprecated old API.
>> * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
>> old API, and a "Evolving" new API.
>> * For some pre-1.0 release (perhaps 0.22), the old API could be
>> deprecated again, and the new API marked as "Stable".
>> * In the 1.0 release it would be possible to remove the old API.
>>
>> Thoughts?
>>
>> Tom
>

I don't have any issue with un-deprecating the old APIs.  I agree if  
changes are needed it's better to mark the new APIs to reflect it.  I  
just hope those changes can be kept as backward compatible as  
possible.  In particular with Job, Pig uses that in some of it's APIs  
that it has declared stable (LoadFunc, StoreFunc).

Alan.

On Apr 22, 2010, at 11:30 PM, Arun C Murthy wrote:

> Alan,
>
> On Apr 22, 2010, at 12:12 PM, Alan Gates wrote:
>
>> Speaking for one power user (Pig) that did move to the new APIs,  
>> moving that interface to evolving is a little unsettling.  Is there  
>> a feel for how much the new API is going to change?
>>
>
> The intent isn't to mark the 'new' apis as 'Evolving' to change them  
> willy-nilly... please don't read it so!
>
> This is just a pragmatic proposal to reflect that the 'old' apis  
> will, for lack of stabilization of new apis, be supported.
>
> Given that, the new apis could mostly be stable, but for Job and  
> Cluster - is that reasonable? This will ensure we send the right  
> message all concerned regarding stability of o.a.h.mapreduce.{Mapper|
> Reducer|...}. Thoughts?
>
> Arun
>
>> Alan.
>>

It sounds like there's no strong objection to un-deprecating the old
API in 0.20 - I'll create a patch for this (see
https://issues.apache.org/jira/browse/MAPREDUCE-1734).

0.21 is less clear cut. However, if the new API were marked as
Evolving, then it's odd, to say the least, if the old API were
deprecated since it would send a confusing message to users. There
seems to be consensus that the new API is Evolving (please comment on
https://issues.apache.org/jira/browse/MAPREDUCE-1623 to discuss
whether all of the new API should be marked Evolving, which the latest
patch does). Indeed, the new API hasn't seen widespread use yet, so it
still seems premature to deprecate the old API in 0.21. I've opened
https://issues.apache.org/jira/browse/MAPREDUCE-1735 where we can
discuss this particular case further.

Cheers,
Tom

On Fri, Apr 23, 2010 at 9:21 AM, Alan Gates <[EMAIL PROTECTED]> wrote:
> I don't have any issue with un-deprecating the old APIs.  I agree if changes
> are needed it's better to mark the new APIs to reflect it.  I just hope
> those changes can be kept as backward compatible as possible.  In particular
> with Job, Pig uses that in some of it's APIs that it has declared stable
> (LoadFunc, StoreFunc).
>
> Alan.
>
> On Apr 22, 2010, at 11:30 PM, Arun C Murthy wrote:
>
>> Alan,
>>
>> On Apr 22, 2010, at 12:12 PM, Alan Gates wrote:
>>
>>> Speaking for one power user (Pig) that did move to the new APIs, moving
>>> that interface to evolving is a little unsettling.  Is there a feel for how
>>> much the new API is going to change?
>>>
>>
>> The intent isn't to mark the 'new' apis as 'Evolving' to change them
>> willy-nilly... please don't read it so!
>>
>> This is just a pragmatic proposal to reflect that the 'old' apis will, for
>> lack of stabilization of new apis, be supported.
>>
>> Given that, the new apis could mostly be stable, but for Job and Cluster -
>> is that reasonable? This will ensure we send the right message all concerned
>> regarding stability of o.a.h.mapreduce.{Mapper|Reducer|...}. Thoughts?
>>
>> Arun
>>
>>> Alan.
>>>
>
>