Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # dev - Avro RPC interop


+
Mike Percy 2012-03-04, 23:56
+
Doug Cutting 2012-03-06, 16:32
Copy link to this message
-
Re: Avro RPC interop
Mike Percy 2012-03-07, 23:18
Hi Scott & Doug,
Thank you for your responses.

I wonder if we couldn't retrofit the Netty implementation with SASL and see how close that gets us to something that looks like it has all the desired features needed to consider standardizing on it, or something close to it.

Regards,
Mike
On Mar 6, 2012, at 8:32 AM, Doug Cutting wrote:

> A proliferation of RPC methods would create interoperability problems.
> So my idea is that Avro should specify just two standard RPC methods:
>  - HTTP; and
>  - a single alternative that permits higher-performance, secure
> implementations.
>
> We've yet to settle on the second.  In particular, we need to agree on
> how to implement:
> - out of order responses; and
> - authentication and encryption.
>
> For out-of-order responses I made a proposal in AVRO-625 that we add an
> optional call ID to request metadata.  The Netty implementation instead
> puts a call ID outside of the framing.  I can see arguments for either
> approach and don't have a strong preference.  We primarily need to
> choose one and implement it consistently.
>
> For authentication and encryption I have proposed SASL.  SASL's
> anonymous mechanism is easy to implement, by just placing a fixed string
> at the front of the first request and response (see the "Anonymous
> Mechanism" section of http://avro.apache.org/docs/current/sasl.html).
> With SASL, implementations can optionally negotiate authentication and
> encryption.
>
> I'd love to see us get this hashed out and included in 1.7.  I'd like to
> see us have at least two implementations before we declare it a standard
> part of the specification.  In Java we could add SASL support to
> NettyTransceiver and/or add out-of-order support to SaslTransceiver.
> Then we'd need someone to implement at least one other compatible
> implementation (Python?, C?).
>
> Doug
>
> On 03/04/2012 03:56 PM, Mike Percy wrote:
>> Hi Avro devs,
>> What is the latest on Avro RPC interoperability? Flume 1.x is currently using the Avro Netty implementation as an RPC protocol, however I am not aware of compatible implementations of this transport in languages other than Java [1] and I wonder if this protocol will ever be compatible [2] with a future standard [3].
>>
>> Would any of you folks recommend running RPC APIs intended for public consumption on top of NettyServer at this time? It seems to be the one with the most active development going on, and appears to have significant interest, which is a good sign. Since there are other RPC transports available as well - SaslSocketServer and HttpServer - I am also wondering what anecdotal experience others can share about these implementations, in terms of speed, interoperability, and maturity.
>>
>> Thanks in advance for your thoughts.
>>
>> Regards,
>> Mike
>>
>> [1] https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%[EMAIL PROTECTED]%3E
>> [2] https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%[EMAIL PROTECTED]%3E
>> [3] https://issues.apache.org/jira/browse/AVRO-341 - specify avro transport in spec
>>
+
Doug Cutting 2012-03-07, 23:31
+
Scott Carey 2012-03-14, 18:42
+
Scott Carey 2012-03-06, 02:41