Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Towards Hadoop 1.0:  Stronger API Compatibility from 0.21 onwards


Copy link to this message
-
Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards
Sanjay Radia 2009-09-28, 18:06

On Sep 28, 2009, at 3:15 AM, Steve Loughran wrote:

> Dhruba Borthakur wrote:
> > It is really nice to have wire-compatibility between clients and  
> servers
> > running different versions of hadoop. The reason we would like  
> this is
> > because we can allow the same client (Hive, etc) submit jobs to two
> > different clusters running different versions of hadoop. But I am  
> not stuck
> > up on the name of the release that supports wire-compatibility, it  
> can be
> > either 1.0  or something later than that.
> > API compatibility  +1
> > Data compatibility +1
> > Job Q compatibility -1Wire compatibility +0
>
>
> That's stability of the job submission network protocol you are  
> looking
> for there.
>   * We need a job submission API that is designed to work over long-
> haul
> links and versions
>   * It does not have to be the same as anything used in-cluster
>   * It does not actually need to run in the JobTracker. An independent
> service bridging the stable long-haul API to an unstable datacentre
> protocol does work, though authentication and user-rights are a  
> troublespot
>

I think you are misinterpreting what Job Q compatibility means.
It is about jobs already in the queue surviving an upgrade across a  
release.

See my initial proposal on Jan 16th:
https://issues.apache.org/jira/browse/HADOOP-5071?focusedCommentId=12664691&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
#action_12664691

Doug argued that it is nice to have but not required for 1.0 - can be  
added later.
sanjay
>
> Similarly, it would be good for a stable long-haul HDFS protocol, such
> as FTP or webdav. Again, no need to build into the namenode .
>
> see http://www.slideshare.net/steve_l/long-haul-hadoop
> and commentary under http://wiki.apache.org/hadoop/BristolHadoopWorkshop
>