This will be my last question for at least the next day or two. :) I just
want to double check my interpretation of the message framing. Assuming that
the client and server have already gone through their handshake. Does this
sound right for a request/response on the wire?

Request:

4 byte length

map of bytes - request metadata

4 byte length

string - message name

for each parameter {

4 byte length

parameter bytes

}

null-terminate

Response:

4 byte length

map of bytes - request metadata

1 byte - 0 or 1 for success

if(false){

4 byte length

bytes containing response

}

else

{

4 byte length

bytes containing error
}
SocketTransceiver seems to support this but it also has this comment stating
it's not standard.

/** A socket-based {@link Transceiver} implementation.  This uses a simple,
 * non-standard wire protocol and is not intended for production services.
*/

A *call* consists of a request message paired with its resulting response or
error message. Requests and responses contain extensible metadata, and both
kinds of messages are framed as described above.

The format of a call request is:

   - *request metadata*, a map with values of type bytes
   - the *message name*, an Avro string, followed by
   - the message *parameters*. Parameters are serialized according to the
   message's request declaration.

The format of a call response is:

   - *response metadata*, a map with values of type bytes
   - a one-byte *error flag* boolean, followed by either:
      - if the error flag is false, the message *response*, serialized per
      the message's response schema.
      - if the error flag is true, the *error*, serialized per the message's
      error union schema.

On 06/07/2010 02:25 PM, Jeremy Custenborder wrote:
> This will be my last question for at least the next day or two. :) I just
> want to double check my interpretation of the message framing.

Framing is mostly orthogonal to payload structure.  It just says that
the bytes that make up a payload are broken into length-prefixed frames.
  Those frames may or may not align with the payload's structure.  When
they do line up, optimizations are possible, so implementations should
try to line them up when sending largish binary objects.

Request payloads are of the form <metadata><messageName><parameters> and
response payloads are of the form <metadata><boolean><errorOrResponse>.
  These may be delivered in one or more frames.  A good framing might be
to use a single frame unless a parameter has a bytes value that's larger
than 4k, in which case that buffer might be transmitted as a distinct
frame, so that it can be potentially processed in zerocopy style.  I.e.,
if it contains file data, then the sendfile system call could be used to
copy data from file to socket.

Does that help?

Doug

Thanks Doug,

That affirms the direction that I was heading. Should I assume that metadata
is unique to each message? Here is what I have done so far for the code
generator portion. Should I worry about generating code that looks like
example 1 where I pass along a dictionary for metadata?  Or should I look at
something like example 2 where I don't pass along the metadata. I could also
do a combination of both.

Example 1
[System.CodeDom.Compiler.GeneratedCodeAttribute("Avro.CodeGen.AvroGen",
"1.0")]
    public interface Simple
    {
        /// <summary>
        /// Send a greeting
        /// </summary>
        string hello(System.Collections.Generic.IDictionary<string, byte[]>
metadata, string greeting);
        /// <summary>
        /// Pretend you're in a cave!
        /// </summary>
        TestRecord echo(System.Collections.Generic.IDictionary<string,
byte[]> metadata, TestRecord record);
        int add(System.Collections.Generic.IDictionary<string, byte[]>
metadata, int arg1, int arg2);
        byte[] echoBytes(System.Collections.Generic.IDictionary<string,
byte[]> metadata, byte[] data);
        /// <summary>
        /// Always throws an error.
        /// </summary>
        void error(System.Collections.Generic.IDictionary<string, byte[]>
metadata);
    }

Example 2
    [System.CodeDom.Compiler.GeneratedCodeAttribute("Avro.CodeGen.AvroGen",
"1.0")]
    public interface Simple
    {
        /// <summary>
        /// Send a greeting
        /// </summary>
        string hello(string greeting);
        /// <summary>
        /// Pretend you're in a cave!
        /// </summary>
        TestRecord echo(TestRecord record);
        int add(int arg1, int arg2);
        byte[] echoBytes(byte[] data);
        /// <summary>
        /// Always throws an error.
        /// </summary>
        void error();
    }

On Mon, Jun 7, 2010 at 5:03 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> On 06/07/2010 02:25 PM, Jeremy Custenborder wrote:
>
>> This will be my last question for at least the next day or two. :) I just
>> want to double check my interpretation of the message framing.
>>
>
> Framing is mostly orthogonal to payload structure.  It just says that the
> bytes that make up a payload are broken into length-prefixed frames.  Those
> frames may or may not align with the payload's structure.  When they do line
> up, optimizations are possible, so implementations should try to line them
> up when sending largish binary objects.
>
> Request payloads are of the form <metadata><messageName><parameters> and
> response payloads are of the form <metadata><boolean><errorOrResponse>.
>  These may be delivered in one or more frames.  A good framing might be to
> use a single frame unless a parameter has a bytes value that's larger than
> 4k, in which case that buffer might be transmitted as a distinct frame, so
> that it can be potentially processed in zerocopy style.  I.e., if it
> contains file data, then the sendfile system call could be used to copy data
> from file to socket.
>
> Does that help?
>
> Doug
>

Hey Jeremy,

The metadata is part of the call request record, which in Avro is a ["null",
{"type": "map", "values": "bytes"}] followed by a "string" followed by a
nameless record of the message parameters (and yes, we should specify this
protocol in Avro; see
https://issues.apache.org/jira/browse/AVRO-341?focusedCommentId=12831204&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12831204
).

Thus I think the latter option looks correct; Doug can tell you
definitively.

Later,
Jeff

On Mon, Jun 7, 2010 at 6:23 PM, Jeremy Custenborder <[EMAIL PROTECTED]
> wrote:

> Thanks Doug,
>
> That affirms the direction that I was heading. Should I assume that
> metadata
> is unique to each message? Here is what I have done so far for the code
> generator portion. Should I worry about generating code that looks like
> example 1 where I pass along a dictionary for metadata?  Or should I look
> at
> something like example 2 where I don't pass along the metadata. I could
> also
> do a combination of both.
>
> Example 1
> [System.CodeDom.Compiler.GeneratedCodeAttribute("Avro.CodeGen.AvroGen",
> "1.0")]
>    public interface Simple
>    {
>        /// <summary>
>        /// Send a greeting
>        /// </summary>
>        string hello(System.Collections.Generic.IDictionary<string, byte[]>
> metadata, string greeting);
>        /// <summary>
>        /// Pretend you're in a cave!
>        /// </summary>
>        TestRecord echo(System.Collections.Generic.IDictionary<string,
> byte[]> metadata, TestRecord record);
>        int add(System.Collections.Generic.IDictionary<string, byte[]>
> metadata, int arg1, int arg2);
>        byte[] echoBytes(System.Collections.Generic.IDictionary<string,
> byte[]> metadata, byte[] data);
>        /// <summary>
>        /// Always throws an error.
>        /// </summary>
>        void error(System.Collections.Generic.IDictionary<string, byte[]>
> metadata);
>    }
>
> Example 2
>    [System.CodeDom.Compiler.GeneratedCodeAttribute("Avro.CodeGen.AvroGen",
> "1.0")]
>    public interface Simple
>    {
>        /// <summary>
>        /// Send a greeting
>        /// </summary>
>        string hello(string greeting);
>        /// <summary>
>        /// Pretend you're in a cave!
>        /// </summary>
>        TestRecord echo(TestRecord record);
>        int add(int arg1, int arg2);
>        byte[] echoBytes(byte[] data);
>        /// <summary>
>        /// Always throws an error.
>        /// </summary>
>        void error();
>     }
>
> On Mon, Jun 7, 2010 at 5:03 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
>
> > On 06/07/2010 02:25 PM, Jeremy Custenborder wrote:
> >
> >> This will be my last question for at least the next day or two. :) I
> just
> >> want to double check my interpretation of the message framing.
> >>
> >
> > Framing is mostly orthogonal to payload structure.  It just says that the
> > bytes that make up a payload are broken into length-prefixed frames.
>  Those
> > frames may or may not align with the payload's structure.  When they do
> line
> > up, optimizations are possible, so implementations should try to line
> them
> > up when sending largish binary objects.
> >
> > Request payloads are of the form <metadata><messageName><parameters> and
> > response payloads are of the form <metadata><boolean><errorOrResponse>.
> >  These may be delivered in one or more frames.  A good framing might be
> to
> > use a single frame unless a parameter has a bytes value that's larger
> than
> > 4k, in which case that buffer might be transmitted as a distinct frame,
> so
> > that it can be potentially processed in zerocopy style.  I.e., if it
> > contains file data, then the sendfile system call could be used to copy
> data
> > from file to socket.
> >
> > Does that help?
> >
> > Doug
> >
>

Example 2 looks right to me.  Metadata is out-of-band data, not provided
directly by client applications or seen directly by server-side
implementations.  It's intended to be useful for implementing call
tracing, performance statistics, session management, etc.  The client
and server runtime might provide hooks that permit plugins to process
metadata.

For example, Java has an RPCPlugin interface:

http://avro.apache.org/docs/current/api/java/org/apache/avro/ipc/RPCPlugin.html

An implementation of this is provided that records server-side RPC
statistics:

http://avro.apache.org/docs/current/api/java/org/apache/avro/ipc/stats/StatsPlugin.html

Doug
On 06/07/2010 06:23 PM, Jeremy Custenborder wrote:
> Thanks Doug,
>
> That affirms the direction that I was heading. Should I assume that metadata
> is unique to each message? Here is what I have done so far for the code
> generator portion. Should I worry about generating code that looks like
> example 1 where I pass along a dictionary for metadata?  Or should I look at
> something like example 2 where I don't pass along the metadata. I could also
> do a combination of both.
>
> Example 1
> [System.CodeDom.Compiler.GeneratedCodeAttribute("Avro.CodeGen.AvroGen",
> "1.0")]
>      public interface Simple
>      {
>          ///<summary>
>          /// Send a greeting
>          ///</summary>
>          string hello(System.Collections.Generic.IDictionary<string, byte[]>
> metadata, string greeting);
>          ///<summary>
>          /// Pretend you're in a cave!
>          ///</summary>
>          TestRecord echo(System.Collections.Generic.IDictionary<string,
> byte[]>  metadata, TestRecord record);
>          int add(System.Collections.Generic.IDictionary<string, byte[]>
> metadata, int arg1, int arg2);
>          byte[] echoBytes(System.Collections.Generic.IDictionary<string,
> byte[]>  metadata, byte[] data);
>          ///<summary>
>          /// Always throws an error.
>          ///</summary>
>          void error(System.Collections.Generic.IDictionary<string, byte[]>
> metadata);
>      }
>
> Example 2
>      [System.CodeDom.Compiler.GeneratedCodeAttribute("Avro.CodeGen.AvroGen",
> "1.0")]
>      public interface Simple
>      {
>          ///<summary>
>          /// Send a greeting
>          ///</summary>
>          string hello(string greeting);
>          ///<summary>
>          /// Pretend you're in a cave!
>          ///</summary>
>          TestRecord echo(TestRecord record);
>          int add(int arg1, int arg2);
>          byte[] echoBytes(byte[] data);
>          ///<summary>
>          /// Always throws an error.
>          ///</summary>
>          void error();
>      }
>
> On Mon, Jun 7, 2010 at 5:03 PM, Doug Cutting<[EMAIL PROTECTED]>  wrote:
>
>> On 06/07/2010 02:25 PM, Jeremy Custenborder wrote:
>>
>>> This will be my last question for at least the next day or two. :) I just
>>> want to double check my interpretation of the message framing.
>>>
>>
>> Framing is mostly orthogonal to payload structure.  It just says that the
>> bytes that make up a payload are broken into length-prefixed frames.  Those
>> frames may or may not align with the payload's structure.  When they do line
>> up, optimizations are possible, so implementations should try to line them
>> up when sending largish binary objects.
>>
>> Request payloads are of the form<metadata><messageName><parameters>  and
>> response payloads are of the form<metadata><boolean><errorOrResponse>.
>>   These may be delivered in one or more frames.  A good framing might be to
>> use a single frame unless a parameter has a bytes value that's larger than
>> 4k, in which case that buffer might be transmitted as a distinct frame, so
>> that it can be potentially processed in zerocopy style.  I.e., if it
>> contains file data, then the sendfile system call could be used to copy data
>> from file to socket.
>>
>> Does that help?
>>
>> Doug
>>
>

Thanks Doug & Jeff,

Jeff
I read link you provided. Does this mean that the current RPC implementation
will be deprecated once AVRO-341 has been finished? What is the time frame
for AVRO-341? Is it planned for the 1.4 release?  For now I'm just going to
get something out that support connection pooling.

Haha a more important question would be when is the 1.4 release planned? The
.net port is coming along nicely and should be ready for an alpha release in
the near future after I do some interop testing. RPC is a feature that I
want to have

Doug
That makes much more sense to me now. Thank you!  The plugin implementation
makes a lot of sense. I'll implement something similar with the .net
version. On a side note I'm thinking I'll be ready for an alpha release
pretty soon. When would you consider accepting a patch?
https://issues.apache.org/jira/browse/AVRO-533 is the issue I am working
against. The patch that is out there is quite old so I'll need to generate a
new one.

J
On Tue, Jun 8, 2010 at 11:37 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> Example 2 looks right to me.  Metadata is out-of-band data, not provided
> directly by client applications or seen directly by server-side
> implementations.  It's intended to be useful for implementing call tracing,
> performance statistics, session management, etc.  The client and server
> runtime might provide hooks that permit plugins to process metadata.
>
> For example, Java has an RPCPlugin interface:
>
>
> http://avro.apache.org/docs/current/api/java/org/apache/avro/ipc/RPCPlugin.html
>
> An implementation of this is provided that records server-side RPC
> statistics:
>
>
> http://avro.apache.org/docs/current/api/java/org/apache/avro/ipc/stats/StatsPlugin.html
>
> Doug
>
>
>
>
>
> On 06/07/2010 06:23 PM, Jeremy Custenborder wrote:
>
>> Thanks Doug,
>>
>> That affirms the direction that I was heading. Should I assume that
>> metadata
>> is unique to each message? Here is what I have done so far for the code
>> generator portion. Should I worry about generating code that looks like
>> example 1 where I pass along a dictionary for metadata?  Or should I look
>> at
>> something like example 2 where I don't pass along the metadata. I could
>> also
>> do a combination of both.
>>
>> Example 1
>> [System.CodeDom.Compiler.GeneratedCodeAttribute("Avro.CodeGen.AvroGen",
>> "1.0")]
>>     public interface Simple
>>     {
>>         ///<summary>
>>         /// Send a greeting
>>         ///</summary>
>>         string hello(System.Collections.Generic.IDictionary<string,
>> byte[]>
>> metadata, string greeting);
>>         ///<summary>
>>         /// Pretend you're in a cave!
>>         ///</summary>
>>         TestRecord echo(System.Collections.Generic.IDictionary<string,
>> byte[]>  metadata, TestRecord record);
>>         int add(System.Collections.Generic.IDictionary<string, byte[]>
>> metadata, int arg1, int arg2);
>>         byte[] echoBytes(System.Collections.Generic.IDictionary<string,
>> byte[]>  metadata, byte[] data);
>>         ///<summary>
>>         /// Always throws an error.
>>         ///</summary>
>>         void error(System.Collections.Generic.IDictionary<string, byte[]>
>> metadata);
>>     }
>>
>> Example 2
>>
>> [System.CodeDom.Compiler.GeneratedCodeAttribute("Avro.CodeGen.AvroGen",
>> "1.0")]
>>     public interface Simple
>>     {
>>         ///<summary>
>>         /// Send a greeting
>>         ///</summary>
>>         string hello(string greeting);
>>         ///<summary>
>>         /// Pretend you're in a cave!
>>         ///</summary>
>>         TestRecord echo(TestRecord record);
>>         int add(int arg1, int arg2);
>>         byte[] echoBytes(byte[] data);
>>         ///<summary>
>>         /// Always throws an error.
>>         ///</summary>
>>         void error();
>>     }
>>
>> On Mon, Jun 7, 2010 at 5:03 PM, Doug Cutting<[EMAIL PROTECTED]>  wrote:
>>
>>  On 06/07/2010 02:25 PM, Jeremy Custenborder wrote:
>>>
>>>  This will be my last question for at least the next day or two. :) I

On 06/08/2010 09:50 AM, Jeremy Custenborder wrote:
> I read link you provided. Does this mean that the current RPC implementation
> will be deprecated once AVRO-341 has been finished?

No.  My belief is that HTTP will continue to be a useful RPC transport
long-term.  It's easy to implement, since most languages already have
HTTP client and server libraries that have often been optimized for
decent performance.  It also facilitates access through firewalls.

HTTP has shortcomings that we hope an Avro-specific transport will
overcome.  It has performance overheads.  It doesn't naturally permit
one-way messages.  Its encryption, authentication and authorization
mechanisms are heavyweight.  Etc.

AVRO-341 is about creating an alternative to HTTP.  Long-term the intent
is that Avro will specify just two transports, HTTP and an
Avro-specific, optimized transport.

> What is the time frame
> for AVRO-341? Is it planned for the 1.4 release?

I doubt it will make the 1.4 release, but we'll see.

> Haha a more important question would be when is the 1.4 release planned? The
> .net port is coming along nicely and should be ready for an alpha release in
> the near future

I'm hoping we'll have enough new functionality to warrant a 1.4 release
sometime this summer.  I'm reluctant to delay releases for particular
features, but, on the other hand, we should probably delay until we have
at least some significant new features.

Browsing Jira for issues with Fix Version of 1.4.0 shows what folks
currently intend for 1.4.0.  http://tinyurl.com/avro140

I'm working on adding MapReduce support.  AVRO-493 is complete, AVRO-512
is nearly so, but more work remains (AVRO-513, AVRO-570, AVRO-567).
Bruce is working on Win32 support for Avro C and other things.  Etc.

> On a side note I'm thinking I'll be ready for an alpha release
> pretty soon. When would you consider accepting a patch?
> https://issues.apache.org/jira/browse/AVRO-533 is the issue I am working
> against. The patch that is out there is quite old so I'll need to generate a
> new one.

The bar is lower for initial implementations: something is better than
nothing.  It should at least build and include tests for the
functionality it provides.  It need not provide datafile or RPC support
before the first commit.  Those can be added in subsequent patches.

C# support would be a wonderful feature for 1.4.

Doug

Thanks for the info. The c# port currently has support to read/write
schemas, serialize data, code generation for schemas and protocols. I'm
working on RPC now then on to file storage. My main usage goals for Avro is
actually cross platform RPC.

I agree with you in most cases HTTP will be just fine. Most people will
never notice the overheads of HTTP. Is there currently a raw TCP
implementation or are the other languages using HTTP as well. I was looking
at DatagramServer in the java source which is why I asked the questions due
to the comment. /** A datagram-based server implementation. This uses a
simple, non-standard wire protocol and is not intended for production
services. */

Is this implemented across each of the languages or just a POC in the java
source? I'm assuming that HTTP is implemented everywhere or in most of the
languages? I am planning on adding support for HTTP in the C# port anyways
but for my usage a lower level TCP implementation would be preferred. If the
spec is not solidified I can just use HTTP for now.

On Tue, Jun 8, 2010 at 12:35 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> On 06/08/2010 09:50 AM, Jeremy Custenborder wrote:
>
>> I read link you provided. Does this mean that the current RPC
>> implementation
>> will be deprecated once AVRO-341 has been finished?
>>
>
> No.  My belief is that HTTP will continue to be a useful RPC transport
> long-term.  It's easy to implement, since most languages already have HTTP
> client and server libraries that have often been optimized for decent
> performance.  It also facilitates access through firewalls.
>
> HTTP has shortcomings that we hope an Avro-specific transport will
> overcome.  It has performance overheads.  It doesn't naturally permit
> one-way messages.  Its encryption, authentication and authorization
> mechanisms are heavyweight.  Etc.
>
> AVRO-341 is about creating an alternative to HTTP.  Long-term the intent is
> that Avro will specify just two transports, HTTP and an Avro-specific,
> optimized transport.
>
>
>  What is the time frame
>> for AVRO-341? Is it planned for the 1.4 release?
>>
>
> I doubt it will make the 1.4 release, but we'll see.
>
>
>  Haha a more important question would be when is the 1.4 release planned?
>> The
>> .net port is coming along nicely and should be ready for an alpha release
>> in
>> the near future
>>
>
> I'm hoping we'll have enough new functionality to warrant a 1.4 release
> sometime this summer.  I'm reluctant to delay releases for particular
> features, but, on the other hand, we should probably delay until we have at
> least some significant new features.
>
> Browsing Jira for issues with Fix Version of 1.4.0 shows what folks
> currently intend for 1.4.0.  http://tinyurl.com/avro140
>
> I'm working on adding MapReduce support.  AVRO-493 is complete, AVRO-512 is
> nearly so, but more work remains (AVRO-513, AVRO-570, AVRO-567). Bruce is
> working on Win32 support for Avro C and other things.  Etc.
>
>
>  On a side note I'm thinking I'll be ready for an alpha release
>> pretty soon. When would you consider accepting a patch?
>> https://issues.apache.org/jira/browse/AVRO-533 is the issue I am working
>> against. The patch that is out there is quite old so I'll need to generate
>> a
>> new one.
>>
>
> The bar is lower for initial implementations: something is better than
> nothing.  It should at least build and include tests for the functionality
> it provides.  It need not provide datafile or RPC support before the first
> commit.  Those can be added in subsequent patches.
>
> C# support would be a wonderful feature for 1.4.
>
> Doug
>

> Is this implemented across each of the languages or just a POC in the java
> source? I'm assuming that HTTP is implemented everywhere or in most of the
> languages? I am planning on adding support for HTTP in the C# port anyways
> but for my usage a lower level TCP implementation would be preferred. If
> the
> spec is not solidified I can just use HTTP for now.
>

Just use HTTP for now. I implemented the socket-based RPC in Python first,
but questions about how things were done there led to the filing of
AVRO-341. We'll get that one hammered out this summer for sure, hopefully
with your input.

Thanks,
Jeff

Ok thanks for clearing that up guys! I'll just implement the HTTP client and
server version for now. Thanks!

On Tue, Jun 8, 2010 at 1:53 PM, Jeff Hammerbacher <[EMAIL PROTECTED]>wrote:

> > Is this implemented across each of the languages or just a POC in the
> java
> > source? I'm assuming that HTTP is implemented everywhere or in most of
> the
> > languages? I am planning on adding support for HTTP in the C# port
> anyways
> > but for my usage a lower level TCP implementation would be preferred. If
> > the
> > spec is not solidified I can just use HTTP for now.
> >
>
> Just use HTTP for now. I implemented the socket-based RPC in Python first,
> but questions about how things were done there led to the filing of
> AVRO-341. We'll get that one hammered out this summer for sure, hopefully
> with your input.
>
> Thanks,
> Jeff
>