Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Towards Hadoop 1.0:  Stronger API Compatibility from 0.21 onwards


Copy link to this message
-
Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

On Sep 28, 2009, at 3:15 AM, Steve Loughran wrote:

> Dhruba Borthakur wrote:
> > It is really nice to have wire-compatibility between clients and  
> servers
> > running different versions of hadoop. The reason we would like  
> this is
> > because we can allow the same client (Hive, etc) submit jobs to two
> > different clusters running different versions of hadoop. But I am  
> not stuck
> > up on the name of the release that supports wire-compatibility, it  
> can be
> > either 1.0  or something later than that.
> > API compatibility  +1
> > Data compatibility +1
> > Job Q compatibility -1Wire compatibility +0
>
>
> That's stability of the job submission network protocol you are  
> looking
> for there.
>   * We need a job submission API that is designed to work over long-
> haul
> links and versions
>   * It does not have to be the same as anything used in-cluster
>   * It does not actually need to run in the JobTracker. An independent
> service bridging the stable long-haul API to an unstable datacentre
> protocol does work, though authentication and user-rights are a  
> troublespot
>

I think you are misinterpreting what Job Q compatibility means.
It is about jobs already in the queue surviving an upgrade across a  
release.

See my initial proposal on Jan 16th:
https://issues.apache.org/jira/browse/HADOOP-5071?focusedCommentId=12664691&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
#action_12664691

Doug argued that it is nice to have but not required for 1.0 - can be  
added later.
sanjay
>
> Similarly, it would be good for a stable long-haul HDFS protocol, such
> as FTP or webdav. Again, no need to build into the namenode .
>
> see http://www.slideshare.net/steve_l/long-haul-hadoop
> and commentary under http://wiki.apache.org/hadoop/BristolHadoopWorkshop
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB