Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Bypassing "handshake" in Responder


Copy link to this message
-
Re: Bypassing "handshake" in Responder
Pankaj Shroff 2013-03-12, 21:01
Quick update on this issue. I think I finally figured out a way to bypass
Avro style "handshake" when employing "custom" format or content type
implementation but still trying to reuse or benefit from the
Serialize/Deserialize capabilities of Avro. Perhaps the following usage is
the "intended" REUSE use case in Avro. However, what I have below
completely bypasses Avro RPC classes in the avro-ipc package. Let me know
if this still mandates a patch proposal on Jira. Basically - I implemented
my own "Responder-Server" combination without relying on reflection based
method invocation of "classic" Avro. Link to source code below:

https://github.com/openrtb/openrtb2x/blob/2.0/demand-side/dsp-core/src/main/java/org/openrtb/dsp/core/DemandSideServer.java

Pankaj

On Wed, Feb 27, 2013 at 4:38 PM, Pankaj Shroff <[EMAIL PROTECTED]> wrote:

> Doug - thanks again.
>
> I agree with you. I have been looking into it for the past few days and it
> seems like this will require quite a bit of refactoring. I will try to
> follow up on Jira with more specific details.
>
> Thanks
> Pankaj
>
>
>
> On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
>
>> Pankaj,
>>
>> Avro RPC is currently specified to always uses the binary encoding
>> (like Avro data files).   This might reasonably be extended to JSON.
>> First we'd need to specify the new wire format.  Probably Avro's
>> framing would not make sense for JSON-encoded RPC over HTTP.  Then
>> we'd need to figure what of the existing Java implementation might be
>> reused or adapted.  At a glance, it doesn't look to me like a few
>> one-line changes would suffice, adding methods where things are
>> hardwired, that rather more substantial changes would be required, but
>> I might be missing something.  If you're interested in pursuing this
>> then please file an issue in Jira where you can propose changes to the
>> specification and implementation.
>>
>> Cheers,
>>
>> Doug
>>
>> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <[EMAIL PROTECTED]>
>> wrote:
>> > I guess my question is more basic - given that this is somewhat
>> specific to
>> > my own use case:
>> >
>> > How does one use other forms of Encoder/Decoder implementations that are
>> > available in the Avro library along with the Avro-Ipc SDK.
>> >
>> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc
>> supports is
>> > the BinaryEncoding
>> >
>> > Pankaj
>> >
>> >
>> >
>> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> Doug
>> >>
>> >> Perhaps you answered a portion of my conundrum in another thread
>> >> (permalink below) - but there is still the handshake and reuse of
>> invocation
>> >> logic question. Let me also think about this a little bit.
>> >>
>> >> Thanks in any case. Avro is a great tool in any case!
>> >>
>> >>
>> >>
>> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
>> >>
>> >>
>> >> Pankaj
>> >>
>> >> [EMAIL PROTECTED]
>> >>
>> >>
>> >>
>> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <[EMAIL PROTECTED]>
>> wrote:
>> >>>
>> >>> This sounds like a different RPC wire format than Avro's.  Avro's
>> >>> Requestor and Responder implement Avro's RPC wire format.  Avro's
>> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
>> >>> implementation of other RPC wire formats that include Avro data.
>> >>> Avro's Transceiver API may or may not be reusable, since it assumes
>> >>> Avro-style framing.  Parts of Requestor and Responder *might* be
>> >>> reusable and some refactoring of those classes *might* make such reuse
>> >>> easier, but there's not that much logic there that's not specific to
>> >>> Avro's wire format, so it might be just as easy to reimplement this
>> >>> layer for a different wire format.  It's hard for me to say without
>> >>> seeing a patch with a proposed refactoring.  Does that make sense?
>> >>>
>> >>> Doug
>> >>
Pankaj Shroff
[EMAIL PROTECTED]