Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Towards Hadoop 1.0:  Stronger API Compatibility from 0.21 onwards

Copy link to this message
Re: Towards Hadoop 1.0:  Stronger API Compatibility from 0.21 onwards

On Aug 28, 2009, at 3:15 PM, Doug Cutting wrote:

> Sanjay Radia wrote:
> > No. The 1.0 proposal was that it included both API and wire  
> compatibility.
> The proposal includes a lot of things, but it's so far just a  
> proposal.
>   There's been no vote to formally define what 1.0 will mean.  In  
> every
> discussion I've heard, from the very beginning of the project, it
> primarily meant API stability.  You've added wire compatibility, data
> stability, security, restart recovery, etc.  These are all very nice
> features to have, essential perhaps in some contexts, but they may nor
> may not be required for 1.0.  I worry that if we keep piling more  
> things
> on, we'll never get to 1.0.
Your email seems to suggest that I have added a whole bunch of ad-hoc  
compatibility features over a period of time. Not true if you
check the record: API, Data Compatibility and Wire compatibility were  
part of my proposal from the early days (see the email thread on this  
subject). Data compatibility is already being supported and I was  
assuming that we are going to continue to support it;
When I added Job Q compatibility, you commented that it was not  
necessary at this stage and I replied on the Jira saying that I agreed  
with you.

Surprised that you feel that wire compatibility was not even on the  
  One of the key motivation for Yahoo developing Avro was wire  
compatibility for hadoop 1.0.
It is okay to propose that we drop wire compatibility from 1.0, but to  
state that this was never part of the discussion is misleading.
As far as voting; there was no vote - so that applies to all aspects  
of the 1.0 proposal.


There are two related but somewhat separate issues here:
  - what should and should not be part of 1.0
  - how do we maintain the stability of the new interfaces that have  
been  marked as stable. (The reason
  this email thread was initiated.)    I think Doug, you are  
suggesting that we addressed this one by calling 22 to be 1.0; a  
reasonable proposal that we should discuss.
     - I am not opposed to discussing something like this but I want  
to better understand your concerns about wire compatibility.
Is it lack of resources in getting wire compatibility or that you  
think it is low priority for the community?
Both Facebook (Dhruba tells me) and Yahoo are suffering badly from the  
lack of wire compatibility -  a major motivaiton
for Yahoo to develop Avro.

To summarize wrt to 1.0:

API compatibility - I suspect all will say yes.
Data compatibility - something that I believe we should continue to  
Job Q compatibility - doug and sanjay have said: not now, good idea  
for later.
Wire compatibility - open question; but my thoughts are:
      With the progress we have made on Avro so far I think there is a  
very good chance to get wire compatibility in 22 which we
can then call 0.99 or 1.0. I think it is worth a shot.
> Doug