Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Bypassing "handshake" in Responder


Copy link to this message
-
Re: Bypassing "handshake" in Responder
Doug - thanks. I think I see what you are saying - but I was trying to
leverage Avro a bit more than perhaps currently designed.

There is a lot of great code in  SpecificResponder that uses JsonDecoder
(for example) to do the data binding between a SpecificData derived object
and its Json representation. The great thing about this is that it
validates against the protocol schema specified (or installed upon
construction of a SpecificResponder object). To be clear, here is an
example:

1) I define Avro protocol file to describe my protocol (.avpr)
2) In the generate-sources phase, avro-maven plugin helps me to generate
Class types for my protocol objects and the interface for the message
exchange (my protocol API). These extend the SpecificRecord object in Avro
IO

3) I write an implementation of the protocol API by implementing the
interface generated by Avro
4) Now I want to use the avro-ipc package to create client and server
classes (Requestor and Responder in avro-ipc)

5) My protocol does not "REQUIRE" Avro RPC framing - surely it CANNOT -
otherwise I would be forcing every client and server implementation to have
a dependency on Avro. They might just want to use direct Jackson or direct
Protobuf or direct Thrift support for encoding/decoding - or they may have
legacy code already in place which they would rather not refactor - just
because of the dependency I created.

6) I therefore need hooks to provide my own implementation of
SpecificDecoder and SpecificEncoder.

There is no way to do this.
If, as you suggest, I extend from the DatumReader/Writer classes directly,
I have to write all the code for codehaus.jackson's ObjectMapper and
JsonParser usage, which is currently nicely encapsulated in SpecificData. I
also have to write the invocation logic (that invokes the method
"respond()" in my implementation instance of my protocol API) which I would
have been just loved to reuse.

Also the hooks mentioned in (6) above would have allowed me to leverage in
the future any other encoder/decoder support Avro may provide (if at all) -
such as Protobuf, Thrift, etc.

In general what I was looking for is an "option" really - to tell Avro's
IPC API whether I am using Avro RPC framing or not! If not, it should just
let me handle the framing bits in a derived class or something and bypass
the handshake.

What do you think? :)

Cheers
Pankaj

--

On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> This sounds like a different RPC wire format than Avro's.  Avro's
> Requestor and Responder implement Avro's RPC wire format.  Avro's
> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
> implementation of other RPC wire formats that include Avro data.
> Avro's Transceiver API may or may not be reusable, since it assumes
> Avro-style framing.  Parts of Requestor and Responder *might* be
> reusable and some refactoring of those classes *might* make such reuse
> easier, but there's not that much logic there that's not specific to
> Avro's wire format, so it might be just as easy to reimplement this
> layer for a different wire format.  It's hard for me to say without
> seeing a patch with a proposed refactoring.  Does that make sense?
>
> Doug
>
> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <[EMAIL PROTECTED]> wrote:
> > Hi
> >
> > We are using Avro for implementing an open source reference
> implementation
> > of the OpenRTB protocol.
> >
> > We have made a design goal to model the protocol using Avro protocol
> files
> > (.avpr) and generate types defined in the protocol schema using Avro .
> The
> > challenge is that the protocol does not necessarily require the use of
> Avro/
> > Binary wire encoding - or even the use of Avro/ RPC context. In fact many
> > counter parties have proprietary implementations supporting either
> Protobuf
> > or Json encoding.
> >
> > Now, there is a Json encoder/decoder in the Avro package but it seems
> that
> > the approach is a "schema-first" approach. The JsonEncoder assumes that

Pankaj Shroff
[EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB