Hadoop 1.0's goal was compatibility on several fronts.
(See https://issues.apache.org/jira/browse/HADOOP-5071) for details.

Due to the amount of work involved, it has been necessary to split  
this work across several  releases prior to 1.0.

Turns out that release 0.21 has a number of Jiras targeted towards API  
and config stability.
Further, in 0.21,  we are tagging interfaces with a classification of  
their intended audience(scope) and their stability
(see HADOOP-5073 for the classification).
Post 1.0 stable interfaces will remain stable (both syntax and  
semantics) according the proposed 1.0 rules.
Hadoop's  pre-1.0 rules allow interfaces to be changed regardless of  
stability as long as one allows 2 releases of deprecation.
(See http://wiki.apache.org/hadoop/Roadmap for the current i.e.  
pre-1.0 rules).

So how do we arrange to maintain that stable interfaces remain stable  
(both syntax and semantics) between 0.21 and 1.0?
I propose that we honor the compatibility of stable interfaces  from  
release 0.21 onwards;
i.e. apply the same post 1.0 rules to pre-1.0 releases.

The actual discussion on what needs to be stable or not belongs inside  
Jira Hadoop-5073, not in this email thread;
  I would like to use this email thread to discuss the proposal of  
honoring  compatibility of stable interfaces prior to 1.0.

Feedback?

sanjay

Sanjay Radia wrote:
> I propose that we honor the compatibility of stable interfaces  from
> release 0.21 onwards;
> i.e. apply the same post 1.0 rules to pre-1.0 releases.

I think that makes 0.21 effectively 1.0.  The hallmark of 1.0 will be
our promise not to change APIs incompatibly until 2.0.  If we make that
promise sooner, then we're just accelerating 1.0.

I understood that the plan was to add inter-release wire compatibility
after 1.0, somewhere in the 1.x series of releases.  So, e.g., if we add
it in 1.1, then a 1.1 and 1.2 cluster would be able to make RPCs to one
another, but 1.0 would not be able to make RPCs to these releases or
vice versa.

Doug

On Aug 28, 2009, at 10:35 AM, Doug Cutting wrote:

> Sanjay Radia wrote:
> > I propose that we honor the compatibility of stable interfaces  from
> > release 0.21 onwards;
> > i.e. apply the same post 1.0 rules to pre-1.0 releases.
>
> I think that makes 0.21 effectively 1.0.  The hallmark of 1.0 will be
> our promise not to change APIs incompatibly until 2.0.  If we make  
> that
> promise sooner, then we're just accelerating 1.0.
>

Not quite. 1.0 includes wire compatibility. We can't call it 1.0  
without wire compatibility.
>
> I understood that the plan was to add inter-release wire compatibility
> after 1.0, somewhere in the 1.x series of releases.
>
No. The 1.0 proposal was that it included both API and wire  
compatibility.

See HADOOP-5071.
> So, e.g., if we add
> it in 1.1, then a 1.1 and 1.2 cluster would be able to make RPCs to  
> one
> another, but 1.0 would not be able to make RPCs to these releases or
> vice versa.
>
> Doug
>

Sanjay Radia wrote:
> No. The 1.0 proposal was that it included both API and wire compatibility.

The proposal includes a lot of things, but it's so far just a proposal.
  There's been no vote to formally define what 1.0 will mean.  In every
discussion I've heard, from the very beginning of the project, it
primarily meant API stability.  You've added wire compatibility, data
stability, security, restart recovery, etc.  These are all very nice
features to have, essential perhaps in some contexts, but they may nor
may not be required for 1.0.  I worry that if we keep piling more things
on, we'll never get to 1.0.

What would be wrong with calling it 1.0 when we have end-user API
stability?  Why would that be a bad thing?

Doug

On Aug 28, 2009, at 3:15 PM, Doug Cutting wrote:

> Sanjay Radia wrote:
> > No. The 1.0 proposal was that it included both API and wire  
> compatibility.
>
> The proposal includes a lot of things, but it's so far just a  
> proposal.
>   There's been no vote to formally define what 1.0 will mean.  In  
> every
> discussion I've heard, from the very beginning of the project, it
> primarily meant API stability.  You've added wire compatibility, data
> stability, security, restart recovery, etc.  These are all very nice
> features to have, essential perhaps in some contexts, but they may nor
> may not be required for 1.0.  I worry that if we keep piling more  
> things
> on, we'll never get to 1.0.
>
Your email seems to suggest that I have added a whole bunch of ad-hoc  
compatibility features over a period of time. Not true if you
check the record: API, Data Compatibility and Wire compatibility were  
part of my proposal from the early days (see the email thread on this  
subject). Data compatibility is already being supported and I was  
assuming that we are going to continue to support it;
When I added Job Q compatibility, you commented that it was not  
necessary at this stage and I replied on the Jira saying that I agreed  
with you.

Surprised that you feel that wire compatibility was not even on the  
table.
  One of the key motivation for Yahoo developing Avro was wire  
compatibility for hadoop 1.0.
It is okay to propose that we drop wire compatibility from 1.0, but to  
state that this was never part of the discussion is misleading.
As far as voting; there was no vote - so that applies to all aspects  
of the 1.0 proposal.

--------------

There are two related but somewhat separate issues here:
  - what should and should not be part of 1.0
  - how do we maintain the stability of the new interfaces that have  
been  marked as stable. (The reason
  this email thread was initiated.)    I think Doug, you are  
suggesting that we addressed this one by calling 22 to be 1.0; a  
reasonable proposal that we should discuss.
     - I am not opposed to discussing something like this but I want  
to better understand your concerns about wire compatibility.
Is it lack of resources in getting wire compatibility or that you  
think it is low priority for the community?
Both Facebook (Dhruba tells me) and Yahoo are suffering badly from the  
lack of wire compatibility -  a major motivaiton
for Yahoo to develop Avro.

To summarize wrt to 1.0:

API compatibility - I suspect all will say yes.
Data compatibility - something that I believe we should continue to  
maintain.
Job Q compatibility - doug and sanjay have said: not now, good idea  
for later.
Wire compatibility - open question; but my thoughts are:
      With the progress we have made on Avro so far I think there is a  
very good chance to get wire compatibility in 22 which we
can then call 0.99 or 1.0. I think it is worth a shot.
>
> Doug
>

Sanjay Radia wrote:
> Both Facebook (Dhruba tells me) and Yahoo are suffering badly from the
> lack of wire compatibility -  a major motivaiton
> for Yahoo to develop Avro.

Indeed.  Wire compatibility is a crucial feature that we should release
as soon as possible.  Perhaps before 1.0 if 1.0 slips, perhaps after if
we discover that it's harder to implement than we anticipate.

> Wire compatibility - open question; but my thoughts are:
>      With the progress we have made on Avro so far I think there is a
> very good chance to get wire compatibility in 22 which we
> can then call 0.99 or 1.0. I think it is worth a shot.

+1 It's certainly worth a shot.

1.0 is fundamentally about being able to upgrade a cluster without
changing application code, i.e., API compatibility.  Wire compatibility
will let folks, e.g., use a single client library version to talk to
clusters running different versions, a wonderful feature, but distinct
from the fundamental goal of 1.0.

In general we should not tie too many features to specific releases in
advance of their implementation, since that causes releases to slip when
features slip.  Rather, we should work hard to implement high-priority
features and release periodically, as features are completed and we are
able to qualify releases.  Long-term API compatibility is a very
high-priority feature.  The first release that has APIs that we think we
can support back-compatibly for perhaps a few years should be called
1.0.  Hopefully that will also have some other high-priority features,
like security, wire compatibility, etc.  But I don't see the purpose of
requiring a specific list of high-priority features besides API
compatibility before we declare 1.0, and doing so could needlessly keep
valuable features from users.

Doug

It is really nice to have wire-compatibility between clients and servers
running different versions of hadoop. The reason we would like this is
because we can allow the same client (Hive, etc) submit jobs to two
different clusters running different versions of hadoop. But I am not stuck
up on the name of the release that supports wire-compatibility, it can be
either 1.0  or something later than that.
API compatibility  +1
Data compatibility +1
Job Q compatibility -1Wire compatibility +0

thanks,
dhruba
On Fri, Sep 25, 2009 at 10:05 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> Sanjay Radia wrote:
>
>> Both Facebook (Dhruba tells me) and Yahoo are suffering badly from the
>> lack of wire compatibility -  a major motivaiton
>> for Yahoo to develop Avro.
>>
>
> Indeed.  Wire compatibility is a crucial feature that we should release as
> soon as possible.  Perhaps before 1.0 if 1.0 slips, perhaps after if we
> discover that it's harder to implement than we anticipate.
>
>  Wire compatibility - open question; but my thoughts are:
>>     With the progress we have made on Avro so far I think there is a very
>> good chance to get wire compatibility in 22 which we
>> can then call 0.99 or 1.0. I think it is worth a shot.
>>
>
> +1 It's certainly worth a shot.
>
> 1.0 is fundamentally about being able to upgrade a cluster without changing
> application code, i.e., API compatibility.  Wire compatibility will let
> folks, e.g., use a single client library version to talk to clusters running
> different versions, a wonderful feature, but distinct from the fundamental
> goal of 1.0.
>
> In general we should not tie too many features to specific releases in
> advance of their implementation, since that causes releases to slip when
> features slip.  Rather, we should work hard to implement high-priority
> features and release periodically, as features are completed and we are able
> to qualify releases.  Long-term API compatibility is a very high-priority
> feature.  The first release that has APIs that we think we can support
> back-compatibly for perhaps a few years should be called 1.0.  Hopefully
> that will also have some other high-priority features, like security, wire
> compatibility, etc.  But I don't see the purpose of requiring a specific
> list of high-priority features besides API compatibility before we declare
> 1.0, and doing so could needlessly keep valuable features from users.
>
> Doug
>

--
Connect to me at http://www.facebook.com/dhruba

On 9/25/09 10:13 AM, "Dhruba Borthakur" <[EMAIL PROTECTED]> wrote:
> It is really nice to have wire-compatibility between clients and servers
> running different versions of hadoop. The reason we would like this is
> because we can allow the same client (Hive, etc) submit jobs to two
> different clusters running different versions of hadoop. But I am not stuck
> up on the name of the release that supports wire-compatibility, it can be
> either 1.0  or something later than that.

To me, the lack of wire compatibility makes will make "Hadoop 1.0" in name
only when in reality it is more like 0.80. :(

On Sep 25, 2009, at 12:03 PM, Allen Wittenauer wrote:

> On 9/25/09 10:13 AM, "Dhruba Borthakur" <[EMAIL PROTECTED]> wrote:
> > It is really nice to have wire-compatibility between clients and  
> servers
> > running different versions of hadoop. The reason we would like  
> this is
> > because we can allow the same client (Hive, etc) submit jobs to two
> > different clusters running different versions of hadoop. But I am  
> not stuck
> > up on the name of the release that supports wire-compatibility, it  
> can be
> > either 1.0  or something later than that.
>
> To me, the lack of wire compatibility makes will make "Hadoop 1.0"  
> in name
> only when in reality it is more like 0.80. :(

My sentiments exactly, though I could learn to live with it ....
>

On 9/25/09 12:44 PM, "Sanjay Radia" <[EMAIL PROTECTED]> wrote:

>
> On Sep 25, 2009, at 12:03 PM, Allen Wittenauer wrote:
>
>> On 9/25/09 10:13 AM, "Dhruba Borthakur" <[EMAIL PROTECTED]> wrote:
>>> It is really nice to have wire-compatibility between clients and
>> servers
>>> running different versions of hadoop. The reason we would like
>> this is
>>> because we can allow the same client (Hive, etc) submit jobs to two
>>> different clusters running different versions of hadoop. But I am
>> not stuck
>>> up on the name of the release that supports wire-compatibility, it
>> can be
>>> either 1.0  or something later than that.
>>
>> To me, the lack of wire compatibility makes will make "Hadoop 1.0"
>> in name
>> only when in reality it is more like 0.80. :(
>
> My sentiments exactly, though I could learn to live with it ....

We just had this discussion today about how to put Hadoop into a production
pipeline.  I was under the impression that 1.0 was going to be wire
compatible too.

This is just so disappointing and, quite frankly, makes 1.0 less than useful
for Real Work.  Great, the APIs don't  change but you still have the same
problems of getting data on/off the grid without upgrading your clients
every time.

To me, without wire compatibility, 1.0 makes me feel pretty "meh; who
cares--we're still going to be in upgrade hell".

Allen Wittenauer wrote:
> This is just so disappointing and, quite frankly, makes 1.0 less than useful
> for Real Work.  Great, the APIs don't  change but you still have the same
> problems of getting data on/off the grid without upgrading your clients
> every time.
>
> To me, without wire compatibility, 1.0 makes me feel pretty "meh; who
> cares--we're still going to be in upgrade hell".

The question is not whether wire compatibility is a good thing.  The
question is whether API compatibility is useless without wire
compatibility and, vice versa, whether wire compatibility is useless
without API compatibility.  They're both valuable features and we should
get both of them out as soon as feasible.

The question is if one slips whether we should we hold the other.  I
don't think we should.  Hence we should not in advance tie a particular
release name to both features.  That's all I'm saying.  I claim that the
1.0 moniker is most strongly tied to API compatibility.  If we can get
wire and other sorts of valuable compatibility into that same release,
then great.  If one comes out earlier or later they're both still
valuable.  But neither needs to block the other.

Doug

On 9/25/09 1:18 PM, "Doug Cutting" <[EMAIL PROTECTED]> wrote:
> The question is not whether wire compatibility is a good thing.  The
> question is whether API compatibility is useless without wire
> compatibility and, vice versa, whether wire compatibility is useless
> without API compatibility.  They're both valuable features and we should
> get both of them out as soon as feasible.
>
> The question is if one slips whether we should we hold the other.  I
> don't think we should.  Hence we should not in advance tie a particular
> release name to both features.  That's all I'm saying.  I claim that the
> 1.0 moniker is most strongly tied to API compatibility.  If we can get
> wire and other sorts of valuable compatibility into that same release,
> then great.  If one comes out earlier or later they're both still
> valuable.  But neither needs to block the other.

Oh, I completely understand.  I'm just throwing in a non-developer's
opinion... because I'm sure I'm not the only one expecting/assuming that 1.0
== completely stable.

Allen Wittenauer wrote:
> Oh, I completely understand.  I'm just throwing in a non-developer's
> opinion... because I'm sure I'm not the only one expecting/assuming that 1.0
> == completely stable.  

If we have to live up to that expectation then we might never release
1.0!  Frankly, I fear the longer we delay a 1.0 release the more we
raise expectations that it will be all things.  Rather, I'd like to have
1.0 to mean just one thing: back-compatible APIs until 2.0, with the
expectation that there will be several 1.x releases between.  We can add
other things to 1.0, which might push it out further, but I don't see
how that helps things much.

Would it be materially better for you if we waited longer before calling
a release 1.0, assuming that the same features are released in the same
order and on the same schedule regardless of the release name?

Doug

On 9/25/09 2:40 PM, "Doug Cutting" <[EMAIL PROTECTED]> wrote:
> Would it be materially better for you if we waited longer before calling
> a release 1.0, assuming that the same features are released in the same
> order and on the same schedule regardless of the release name?

Yes.

There is something magic to managerial types when you say "This is not 1.0"
that make them realize that things are far from reliable/stable/practical
from an operations perspective.  When you say "1.0" the
tool/product/whatever those expectations are way higher.

Dhruba Borthakur wrote:
> It is really nice to have wire-compatibility between clients and servers
> running different versions of hadoop. The reason we would like this is
> because we can allow the same client (Hive, etc) submit jobs to two
> different clusters running different versions of hadoop. But I am not stuck
> up on the name of the release that supports wire-compatibility, it can be
> either 1.0  or something later than that.
> API compatibility  +1
> Data compatibility +1
> Job Q compatibility -1Wire compatibility +0
That's stability of the job submission network protocol you are looking
for there.
  * We need a job submission API that is designed to work over long-haul
links and versions
  * It does not have to be the same as anything used in-cluster
  * It does not actually need to run in the JobTracker. An independent
service bridging the stable long-haul API to an unstable datacentre
protocol does work, though authentication and user-rights are a troublespot

Similarly, it would be good for a stable long-haul HDFS protocol, such
as FTP or webdav. Again, no need to build into the namenode .

see http://www.slideshare.net/steve_l/long-haul-hadoop
and commentary under http://wiki.apache.org/hadoop/BristolHadoopWorkshop

HBase and similar HDFS clients could benefit from a (high) performant
stable datacenter network protocol that is built into the namenode and
datanodes. Then we could decouple from Hadoop versioning and release
cycle. HDFS could decouple from core, etc.

Whatever stable network protocol is devised, if any, of course should
perform as well if not better than the current one. A stable but lower
performing option, unfortunately, would be excluded from consideration
right away.

HBase is a bit of a special case currently perhaps in that its access
pattern is random read/write and it may be only a handful of clients
like that. However if HDFS is positioned as a product in its own right,
which I believe is the case since the split, there may be many other
potential users of it -- for all of its benefits -- given a stable
wire format that enables decoupled development.

API compatibility  +1
Data compatibility +1
Wire compatibility +1

Best regards,

Andrew Purtell
Committing Member, HBase Project: hbase.org

________________________________
From: Steve Loughran <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, September 28, 2009 3:15:09 AM
Subject: Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

Dhruba Borthakur wrote:
> It is really nice to have wire-compatibility between clients and servers
> running different versions of hadoop. The reason we would like this is
> because we can allow the same client (Hive, etc) submit jobs to two
> different clusters running different versions of hadoop. But I am not stuck
> up on the name of the release that supports wire-compatibility, it can be
> either 1.0  or something later than that.
> API compatibility  +1
> Data compatibility +1
> Job Q compatibility -1Wire compatibility +0
That's stability of the job submission network protocol you are looking for there.
* We need a job submission API that is designed to work over long-haul links and versions
* It does not have to be the same as anything used in-cluster
* It does not actually need to run in the JobTracker. An independent service bridging the stable long-haul API to an unstable datacentre protocol does work, though authentication and user-rights are a troublespot

Similarly, it would be good for a stable long-haul HDFS protocol, such as FTP or webdav. Again, no need to build into the namenode .

see http://www.slideshare.net/steve_l/long-haul-hadoop
and commentary under http://wiki.apache.org/hadoop/BristolHadoopWorkshop

      

On Sep 28, 2009, at 3:15 AM, Steve Loughran wrote:

> Dhruba Borthakur wrote:
> > It is really nice to have wire-compatibility between clients and  
> servers
> > running different versions of hadoop. The reason we would like  
> this is
> > because we can allow the same client (Hive, etc) submit jobs to two
> > different clusters running different versions of hadoop. But I am  
> not stuck
> > up on the name of the release that supports wire-compatibility, it  
> can be
> > either 1.0  or something later than that.
> > API compatibility  +1
> > Data compatibility +1
> > Job Q compatibility -1Wire compatibility +0
>
>
> That's stability of the job submission network protocol you are  
> looking
> for there.
>   * We need a job submission API that is designed to work over long-
> haul
> links and versions
>   * It does not have to be the same as anything used in-cluster
>   * It does not actually need to run in the JobTracker. An independent
> service bridging the stable long-haul API to an unstable datacentre
> protocol does work, though authentication and user-rights are a  
> troublespot
>

I think you are misinterpreting what Job Q compatibility means.
It is about jobs already in the queue surviving an upgrade across a  
release.

See my initial proposal on Jan 16th:
https://issues.apache.org/jira/browse/HADOOP-5071?focusedCommentId=12664691&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
#action_12664691

Doug argued that it is nice to have but not required for 1.0 - can be  
added later.
sanjay
>
> Similarly, it would be good for a stable long-haul HDFS protocol, such
> as FTP or webdav. Again, no need to build into the namenode .
>
> see http://www.slideshare.net/steve_l/long-haul-hadoop
> and commentary under http://wiki.apache.org/hadoop/BristolHadoopWorkshop
>

I think we should not require Job Q compatibility for 1.0 release.

thanks,
dhruba
On Mon, Sep 28, 2009 at 11:06 AM, Sanjay Radia <[EMAIL PROTECTED]> wrote:

>
> On Sep 28, 2009, at 3:15 AM, Steve Loughran wrote:
>
>  Dhruba Borthakur wrote:
>> > It is really nice to have wire-compatibility between clients and servers
>> > running different versions of hadoop. The reason we would like this is
>> > because we can allow the same client (Hive, etc) submit jobs to two
>> > different clusters running different versions of hadoop. But I am not
>> stuck
>> > up on the name of the release that supports wire-compatibility, it can
>> be
>> > either 1.0  or something later than that.
>> > API compatibility  +1
>> > Data compatibility +1
>> > Job Q compatibility -1Wire compatibility +0
>>
>>
>> That's stability of the job submission network protocol you are looking
>> for there.
>>  * We need a job submission API that is designed to work over long-haul
>> links and versions
>>  * It does not have to be the same as anything used in-cluster
>>  * It does not actually need to run in the JobTracker. An independent
>> service bridging the stable long-haul API to an unstable datacentre
>> protocol does work, though authentication and user-rights are a
>> troublespot
>>
>>
>
>
> I think you are misinterpreting what Job Q compatibility means.
> It is about jobs already in the queue surviving an upgrade across a
> release.
>
> See my initial proposal on Jan 16th:
>
> https://issues.apache.org/jira/browse/HADOOP-5071?focusedCommentId=12664691&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
> #action_12664691
>
> Doug argued that it is nice to have but not required for 1.0 - can be added
> later.
>
>
> sanjay
>
>
>> Similarly, it would be good for a stable long-haul HDFS protocol, such
>> as FTP or webdav. Again, no need to build into the namenode .
>>
>> see http://www.slideshare.net/steve_l/long-haul-hadoop
>> and commentary under http://wiki.apache.org/hadoop/BristolHadoopWorkshop
>>
>>
>
--
Connect to me at http://www.facebook.com/dhruba