Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Bypassing "handshake" in Responder

Copy link to this message
Re: Bypassing "handshake" in Responder
Doug - thanks. I think I see what you are saying - but I was trying to
leverage Avro a bit more than perhaps currently designed.

There is a lot of great code in  SpecificResponder that uses JsonDecoder
(for example) to do the data binding between a SpecificData derived object
and its Json representation. The great thing about this is that it
validates against the protocol schema specified (or installed upon
construction of a SpecificResponder object). To be clear, here is an

1) I define Avro protocol file to describe my protocol (.avpr)
2) In the generate-sources phase, avro-maven plugin helps me to generate
Class types for my protocol objects and the interface for the message
exchange (my protocol API). These extend the SpecificRecord object in Avro

3) I write an implementation of the protocol API by implementing the
interface generated by Avro
4) Now I want to use the avro-ipc package to create client and server
classes (Requestor and Responder in avro-ipc)

5) My protocol does not "REQUIRE" Avro RPC framing - surely it CANNOT -
otherwise I would be forcing every client and server implementation to have
a dependency on Avro. They might just want to use direct Jackson or direct
Protobuf or direct Thrift support for encoding/decoding - or they may have
legacy code already in place which they would rather not refactor - just
because of the dependency I created.

6) I therefore need hooks to provide my own implementation of
SpecificDecoder and SpecificEncoder.

There is no way to do this.
If, as you suggest, I extend from the DatumReader/Writer classes directly,
I have to write all the code for codehaus.jackson's ObjectMapper and
JsonParser usage, which is currently nicely encapsulated in SpecificData. I
also have to write the invocation logic (that invokes the method
"respond()" in my implementation instance of my protocol API) which I would
have been just loved to reuse.

Also the hooks mentioned in (6) above would have allowed me to leverage in
the future any other encoder/decoder support Avro may provide (if at all) -
such as Protobuf, Thrift, etc.

In general what I was looking for is an "option" really - to tell Avro's
IPC API whether I am using Avro RPC framing or not! If not, it should just
let me handle the framing bits in a derived class or something and bypass
the handshake.

What do you think? :)



On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> This sounds like a different RPC wire format than Avro's.  Avro's
> Requestor and Responder implement Avro's RPC wire format.  Avro's
> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
> implementation of other RPC wire formats that include Avro data.
> Avro's Transceiver API may or may not be reusable, since it assumes
> Avro-style framing.  Parts of Requestor and Responder *might* be
> reusable and some refactoring of those classes *might* make such reuse
> easier, but there's not that much logic there that's not specific to
> Avro's wire format, so it might be just as easy to reimplement this
> layer for a different wire format.  It's hard for me to say without
> seeing a patch with a proposed refactoring.  Does that make sense?
> Doug
> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <[EMAIL PROTECTED]> wrote:
> > Hi
> >
> > We are using Avro for implementing an open source reference
> implementation
> > of the OpenRTB protocol.
> >
> > We have made a design goal to model the protocol using Avro protocol
> files
> > (.avpr) and generate types defined in the protocol schema using Avro .
> The
> > challenge is that the protocol does not necessarily require the use of
> Avro/
> > Binary wire encoding - or even the use of Avro/ RPC context. In fact many
> > counter parties have proprietary implementations supporting either
> Protobuf
> > or Json encoding.
> >
> > Now, there is a Json encoder/decoder in the Avro package but it seems
> that
> > the approach is a "schema-first" approach. The JsonEncoder assumes that

Pankaj Shroff