Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # general >> HTTP transport?


+
Doug Cutting 2009-09-11, 21:41
+
Patrick Hunt 2009-11-12, 22:22
+
Scott Carey 2009-09-26, 00:36
+
Doug Cutting 2009-09-28, 17:01
+
Owen OMalley 2009-09-28, 17:59
+
Doug Cutting 2009-09-28, 22:42
+
Sanjay Radia 2009-09-29, 18:52
+
Doug Cutting 2009-09-29, 19:43
+
stack 2009-09-29, 20:38
+
Doug Cutting 2009-09-29, 21:08
+
stack 2009-09-29, 21:57
+
Doug Cutting 2009-09-29, 23:17
+
Devaraj Das 2009-09-29, 23:57
+
Scott Carey 2009-09-30, 01:37
+
Eric Sammer 2009-10-05, 20:43
+
Ryan Rawson 2009-10-05, 20:47
+
Eric Sammer 2009-10-05, 20:53
+
Scott Carey 2009-10-06, 02:59
+
Eric Sammer 2009-10-06, 03:15
+
Scott Carey 2009-10-06, 03:30
+
Owen OMalley 2009-10-08, 22:10
+
Doug Cutting 2009-10-09, 17:49
+
Sanjay Radia 2009-10-09, 18:13
+
Doug Cutting 2009-10-09, 19:56
+
Kan Zhang 2009-10-14, 01:59
+
Doug Cutting 2009-10-14, 16:37
+
Kan Zhang 2009-11-06, 19:15
+
Doug Cutting 2009-11-06, 21:06
+
Kan Zhang 2009-10-14, 17:45
+
Scott Carey 2009-10-11, 01:11
+
Scott Carey 2009-10-06, 03:19
+
Scott Carey 2009-09-30, 03:06
+
Ryan Rawson 2009-09-30, 03:20
+
Raghu Angadi 2009-09-29, 22:11
+
Doug Cutting 2009-09-29, 23:35
+
Sanjay Radia 2009-09-30, 23:04
+
Sanjay Radia 2009-10-05, 16:41
+
Doug Cutting 2009-10-05, 23:48
Copy link to this message
-
Re: HTTP transport?

On Sep 11, 2009, at 2:41 PM, Doug Cutting wrote:

> I'm considering an HTTP-based transport for Avro as the preferred,
> high-performance option.
>
> HTTP has lots of advantages.  In particular, it already has
>   - lots of authentication, authorization and encryption support;
>   - highly optimized servers;
>   - monitoring, logging, etc.
>

Q. Is this to replace the client-DN  data-transfer protocol or for ALL  
Hadoop rpc?

Q. Was authentication one of your main motivation?
The current plans for authentication is centered around kerberos.
HTTP does not fit in too well in that picture.
sanjay
>
> Tomcat and other servlet containers support async NIO, where a  
> thread is
> not required per connection.  A servlet can process bulk data with a
> single copy to and from the socket (bypassing stream buffers).  Calls
> can be multiplexed over a single HTTP connection using Comet events.
>
> http://tomcat.apache.org/tomcat-6.0-doc/aio.html
>
> Zero copy is not an option for servlets that generate arbitrary data,
> but one can specify a file/start/length tuple and Tomcat will use
> sendfile to write the response.  That means that while HDFS datanode
> file reads could not be done via RPC, they could be done via HTTP with
> zero-copy.  If authentication and authorization are already done in  
> the
> HTTP server, this may not be a big loss.  The HDFS client might make  
> two
> HTTP requests, one to read a files data, and another to read its
> checksums.  The server would then stream the entire block to the  
> client
> using sendfile, using TCP flow control as today.
>
> Thoughts?
>
> Doug
>

+
Ryan Rawson 2009-10-05, 20:57
+
Eric Sammer 2009-10-05, 21:13
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB