Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # general >> HTTP transport?


+
Doug Cutting 2009-09-11, 21:41
+
Patrick Hunt 2009-11-12, 22:22
+
Scott Carey 2009-09-26, 00:36
+
Doug Cutting 2009-09-28, 17:01
+
Owen OMalley 2009-09-28, 17:59
+
Doug Cutting 2009-09-28, 22:42
+
Sanjay Radia 2009-09-29, 18:52
+
Doug Cutting 2009-09-29, 19:43
+
stack 2009-09-29, 20:38
+
Doug Cutting 2009-09-29, 21:08
+
stack 2009-09-29, 21:57
+
Doug Cutting 2009-09-29, 23:17
+
Devaraj Das 2009-09-29, 23:57
+
Scott Carey 2009-09-30, 01:37
+
Eric Sammer 2009-10-05, 20:43
+
Ryan Rawson 2009-10-05, 20:47
+
Eric Sammer 2009-10-05, 20:53
+
Scott Carey 2009-10-06, 02:59
+
Eric Sammer 2009-10-06, 03:15
+
Scott Carey 2009-10-06, 03:30
+
Owen OMalley 2009-10-08, 22:10
+
Doug Cutting 2009-10-09, 17:49
+
Sanjay Radia 2009-10-09, 18:13
+
Doug Cutting 2009-10-09, 19:56
Copy link to this message
-
Re: HTTP transport?

On 10/9/09 12:56 PM, "Doug Cutting" <[EMAIL PROTECTED]> wrote:

> Sanjay Radia wrote:
>> Will the RPC over HTTP be transparent so that that we can replace with a
>> different layer if needed?
>
> Yes.
>
>> My worry was the separation of data and checksums; someone had mentioned
>> that one could do this over 2 RPCs - that is not transparent.
>
> That was suggested as a possibility if we did not want to use RPC for
> data, but rather raw HTTP, e.g., with a separate URL per block.  The
> zerocopy support built into most HTTP servers only supports entire
> responses from a single file, so if we wanted to take advantage of these
> zerocopy implementations we'd not use RPC for block access, but could
> use HTTP and hence share security, etc.  Using raw HTTP for block access
> might also perform better, since it can use TCP flow control, rather
> than RPC call/response.  In my microbenchmarks, RPC call/response was
> fast enough to easily saturate disks and networks, so that might be
> moot, although RPC call/response for file data may use more CPU than
> we'd like.  With our own transport implementation we could get RPC
> call/response to use zerocopy for file data.
>

One problem I see with using HTTP is that it's expensive to provide data
encryption. We're currently adding 2 authentication mechanisms (Kerberos and
DIGEST-MD5) to our existing RPC. Both of them can provide data encryption
for subsequent communication over the authenticated channel. However, when
similar authentication mechanisms are specified for HTTP (SPNEGO and HTTP
DIGEST, respectively), they don't provide data encryption (correct me if I'm
wrong). For data encryption over HTTP, one has to use SSL, which is
expensive.

Kan
+
Doug Cutting 2009-10-14, 16:37
+
Kan Zhang 2009-11-06, 19:15
+
Doug Cutting 2009-11-06, 21:06
+
Kan Zhang 2009-10-14, 17:45
+
Scott Carey 2009-10-11, 01:11
+
Scott Carey 2009-10-06, 03:19
+
Scott Carey 2009-09-30, 03:06
+
Ryan Rawson 2009-09-30, 03:20
+
Raghu Angadi 2009-09-29, 22:11
+
Doug Cutting 2009-09-29, 23:35
+
Sanjay Radia 2009-09-30, 23:04
+
Sanjay Radia 2009-10-05, 16:41
+
Doug Cutting 2009-10-05, 23:48
+
Sanjay Radia 2009-09-28, 20:13
+
Ryan Rawson 2009-10-05, 20:57
+
Eric Sammer 2009-10-05, 21:13
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB