Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> Towards Hadoop 1.0:  Stronger API Compatibility from 0.21 onwards


+
Sanjay Radia 2009-08-28, 00:43
+
Doug Cutting 2009-08-28, 17:35
+
Sanjay Radia 2009-08-28, 21:52
+
Doug Cutting 2009-08-28, 22:15
+
Sanjay Radia 2009-09-25, 15:28
+
Doug Cutting 2009-09-25, 17:05
+
Dhruba Borthakur 2009-09-25, 17:13
+
Allen Wittenauer 2009-09-25, 19:03
+
Sanjay Radia 2009-09-25, 19:44
+
Allen Wittenauer 2009-09-25, 19:50
+
Doug Cutting 2009-09-25, 20:18
+
Allen Wittenauer 2009-09-25, 20:55
+
Doug Cutting 2009-09-25, 21:40
+
Allen Wittenauer 2009-09-25, 21:51
+
Steve Loughran 2009-09-28, 10:15
+
Andrew Purtell 2009-09-28, 17:25
Copy link to this message
-
Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

On Sep 28, 2009, at 3:15 AM, Steve Loughran wrote:

> Dhruba Borthakur wrote:
> > It is really nice to have wire-compatibility between clients and  
> servers
> > running different versions of hadoop. The reason we would like  
> this is
> > because we can allow the same client (Hive, etc) submit jobs to two
> > different clusters running different versions of hadoop. But I am  
> not stuck
> > up on the name of the release that supports wire-compatibility, it  
> can be
> > either 1.0  or something later than that.
> > API compatibility  +1
> > Data compatibility +1
> > Job Q compatibility -1Wire compatibility +0
>
>
> That's stability of the job submission network protocol you are  
> looking
> for there.
>   * We need a job submission API that is designed to work over long-
> haul
> links and versions
>   * It does not have to be the same as anything used in-cluster
>   * It does not actually need to run in the JobTracker. An independent
> service bridging the stable long-haul API to an unstable datacentre
> protocol does work, though authentication and user-rights are a  
> troublespot
>

I think you are misinterpreting what Job Q compatibility means.
It is about jobs already in the queue surviving an upgrade across a  
release.

See my initial proposal on Jan 16th:
https://issues.apache.org/jira/browse/HADOOP-5071?focusedCommentId=12664691&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
#action_12664691

Doug argued that it is nice to have but not required for 1.0 - can be  
added later.
sanjay
>
> Similarly, it would be good for a stable long-haul HDFS protocol, such
> as FTP or webdav. Again, no need to build into the namenode .
>
> see http://www.slideshare.net/steve_l/long-haul-hadoop
> and commentary under http://wiki.apache.org/hadoop/BristolHadoopWorkshop
>

+
Dhruba Borthakur 2009-09-28, 19:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB