Hi Avro devs,
What is the latest on Avro RPC interoperability? Flume 1.x is currently using the Avro Netty implementation as an RPC protocol, however I am not aware of compatible implementations of this transport in languages other than Java [1] and I wonder if this protocol will ever be compatible [2] with a future standard [3].

Would any of you folks recommend running RPC APIs intended for public consumption on top of NettyServer at this time? It seems to be the one with the most active development going on, and appears to have significant interest, which is a good sign. Since there are other RPC transports available as well - SaslSocketServer and HttpServer - I am also wondering what anecdotal experience others can share about these implementations, in terms of speed, interoperability, and maturity.

Thanks in advance for your thoughts.

Regards,
Mike

[1] https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%[EMAIL PROTECTED]%3E
[2] https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%[EMAIL PROTECTED]%3E
[3] https://issues.apache.org/jira/browse/AVRO-341 - specify avro transport in spec

I think it is reasonable to push for the consensus on these issues, at
least with respect to spec and direction, for 1.7.0.  This will require a
push amongst us volunteers to complete a concrete implementation and spec.
 Cross-language asynchronous, out-of-order RPC is an important piece of
the Avro puzzle.

This is one of the larger open issues in the community.  I am willing to
host an Avro developer meeting to discuss this and other nominated issues
if the community feels that would be useful.  (Downtown San Francisco /
google+ video).

-Scott Carey

On 3/4/12 3:56 PM, "Mike Percy" <[EMAIL PROTECTED]> wrote:

>Hi Avro devs,
>What is the latest on Avro RPC interoperability? Flume 1.x is currently
>using the Avro Netty implementation as an RPC protocol, however I am not
>aware of compatible implementations of this transport in languages other
>than Java [1] and I wonder if this protocol will ever be compatible [2]
>with a future standard [3].
>
>Would any of you folks recommend running RPC APIs intended for public
>consumption on top of NettyServer at this time? It seems to be the one
>with the most active development going on, and appears to have
>significant interest, which is a good sign. Since there are other RPC
>transports available as well - SaslSocketServer and HttpServer - I am
>also wondering what anecdotal experience others can share about these
>implementations, in terms of speed, interoperability, and maturity.
>
>Thanks in advance for your thoughts.
>
>Regards,
>Mike
>
>[1]
>https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%3C4F1DC9EC
>[EMAIL PROTECTED]%3E
>[2]
>https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%3C4F1A09B9
>[EMAIL PROTECTED]%3E
>[3] https://issues.apache.org/jira/browse/AVRO-341 - specify avro
>transport in spec
>

A proliferation of RPC methods would create interoperability problems.
So my idea is that Avro should specify just two standard RPC methods:
  - HTTP; and
  - a single alternative that permits higher-performance, secure
implementations.

We've yet to settle on the second.  In particular, we need to agree on
how to implement:
 - out of order responses; and
 - authentication and encryption.

For out-of-order responses I made a proposal in AVRO-625 that we add an
optional call ID to request metadata.  The Netty implementation instead
puts a call ID outside of the framing.  I can see arguments for either
approach and don't have a strong preference.  We primarily need to
choose one and implement it consistently.

For authentication and encryption I have proposed SASL.  SASL's
anonymous mechanism is easy to implement, by just placing a fixed string
at the front of the first request and response (see the "Anonymous
Mechanism" section of http://avro.apache.org/docs/current/sasl.html).
With SASL, implementations can optionally negotiate authentication and
encryption.

I'd love to see us get this hashed out and included in 1.7.  I'd like to
see us have at least two implementations before we declare it a standard
part of the specification.  In Java we could add SASL support to
NettyTransceiver and/or add out-of-order support to SaslTransceiver.
Then we'd need someone to implement at least one other compatible
implementation (Python?, C?).

Doug

On 03/04/2012 03:56 PM, Mike Percy wrote:
> Hi Avro devs,
> What is the latest on Avro RPC interoperability? Flume 1.x is currently using the Avro Netty implementation as an RPC protocol, however I am not aware of compatible implementations of this transport in languages other than Java [1] and I wonder if this protocol will ever be compatible [2] with a future standard [3].
>
> Would any of you folks recommend running RPC APIs intended for public consumption on top of NettyServer at this time? It seems to be the one with the most active development going on, and appears to have significant interest, which is a good sign. Since there are other RPC transports available as well - SaslSocketServer and HttpServer - I am also wondering what anecdotal experience others can share about these implementations, in terms of speed, interoperability, and maturity.
>
> Thanks in advance for your thoughts.
>
> Regards,
> Mike
>
> [1] https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%[EMAIL PROTECTED]%3E
> [2] https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%[EMAIL PROTECTED]%3E
> [3] https://issues.apache.org/jira/browse/AVRO-341 - specify avro transport in spec
>

Hi Scott & Doug,
Thank you for your responses.

I wonder if we couldn't retrofit the Netty implementation with SASL and see how close that gets us to something that looks like it has all the desired features needed to consider standardizing on it, or something close to it.

Regards,
Mike
On Mar 6, 2012, at 8:32 AM, Doug Cutting wrote:

> A proliferation of RPC methods would create interoperability problems.
> So my idea is that Avro should specify just two standard RPC methods:
>  - HTTP; and
>  - a single alternative that permits higher-performance, secure
> implementations.
>
> We've yet to settle on the second.  In particular, we need to agree on
> how to implement:
> - out of order responses; and
> - authentication and encryption.
>
> For out-of-order responses I made a proposal in AVRO-625 that we add an
> optional call ID to request metadata.  The Netty implementation instead
> puts a call ID outside of the framing.  I can see arguments for either
> approach and don't have a strong preference.  We primarily need to
> choose one and implement it consistently.
>
> For authentication and encryption I have proposed SASL.  SASL's
> anonymous mechanism is easy to implement, by just placing a fixed string
> at the front of the first request and response (see the "Anonymous
> Mechanism" section of http://avro.apache.org/docs/current/sasl.html).
> With SASL, implementations can optionally negotiate authentication and
> encryption.
>
> I'd love to see us get this hashed out and included in 1.7.  I'd like to
> see us have at least two implementations before we declare it a standard
> part of the specification.  In Java we could add SASL support to
> NettyTransceiver and/or add out-of-order support to SaslTransceiver.
> Then we'd need someone to implement at least one other compatible
> implementation (Python?, C?).
>
> Doug
>
> On 03/04/2012 03:56 PM, Mike Percy wrote:
>> Hi Avro devs,
>> What is the latest on Avro RPC interoperability? Flume 1.x is currently using the Avro Netty implementation as an RPC protocol, however I am not aware of compatible implementations of this transport in languages other than Java [1] and I wonder if this protocol will ever be compatible [2] with a future standard [3].
>>
>> Would any of you folks recommend running RPC APIs intended for public consumption on top of NettyServer at this time? It seems to be the one with the most active development going on, and appears to have significant interest, which is a good sign. Since there are other RPC transports available as well - SaslSocketServer and HttpServer - I am also wondering what anecdotal experience others can share about these implementations, in terms of speed, interoperability, and maturity.
>>
>> Thanks in advance for your thoughts.
>>
>> Regards,
>> Mike
>>
>> [1] https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%[EMAIL PROTECTED]%3E
>> [2] https://mail-archives.apache.org/mod_mbox/avro-dev/201201.mbox/%[EMAIL PROTECTED]%3E
>> [3] https://issues.apache.org/jira/browse/AVRO-341 - specify avro transport in spec
>>

On 03/07/2012 03:18 PM, Mike Percy wrote:
> I wonder if we couldn't retrofit the Netty implementation with SASL
> and see how close that gets us to something that looks like it has
> all the desired features needed to consider standardizing on it, or
> something close to it.

That sounds like a fine plan.

Doug

Another interesting development to consider is that SPDY is starting to
gain wide adoption.
http://wiki.eclipse.org/Jetty/Feature/SPDY

Free load balancers (nginx, apache, HAProxy) either have it or are
planning support for it.  As a replacement for HTTP it supports
out-of-order requests and server initiated requests.  Since load balancers
and proxies will support SPDY over time, and will not likely support a
proprietary avro protocol, Avro over HTTP/SPDY will have its uses.
However, the above developments do not reduce the need for Avro to have
standardized secure socket based asynchronous RPC spec.

On 3/7/12 3:31 PM, "Doug Cutting" <[EMAIL PROTECTED]> wrote:

>On 03/07/2012 03:18 PM, Mike Percy wrote:
>> I wonder if we couldn't retrofit the Netty implementation with SASL
>> and see how close that gets us to something that looks like it has
>> all the desired features needed to consider standardizing on it, or
>> something close to it.
>
>That sounds like a fine plan.
>
>Doug