Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Bypassing "handshake" in Responder


Copy link to this message
-
Re: Bypassing "handshake" in Responder
Quick update on this issue. I think I finally figured out a way to bypass
Avro style "handshake" when employing "custom" format or content type
implementation but still trying to reuse or benefit from the
Serialize/Deserialize capabilities of Avro. Perhaps the following usage is
the "intended" REUSE use case in Avro. However, what I have below
completely bypasses Avro RPC classes in the avro-ipc package. Let me know
if this still mandates a patch proposal on Jira. Basically - I implemented
my own "Responder-Server" combination without relying on reflection based
method invocation of "classic" Avro. Link to source code below:

https://github.com/openrtb/openrtb2x/blob/2.0/demand-side/dsp-core/src/main/java/org/openrtb/dsp/core/DemandSideServer.java

Pankaj

On Wed, Feb 27, 2013 at 4:38 PM, Pankaj Shroff <[EMAIL PROTECTED]> wrote:

> Doug - thanks again.
>
> I agree with you. I have been looking into it for the past few days and it
> seems like this will require quite a bit of refactoring. I will try to
> follow up on Jira with more specific details.
>
> Thanks
> Pankaj
>
>
>
> On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
>
>> Pankaj,
>>
>> Avro RPC is currently specified to always uses the binary encoding
>> (like Avro data files).   This might reasonably be extended to JSON.
>> First we'd need to specify the new wire format.  Probably Avro's
>> framing would not make sense for JSON-encoded RPC over HTTP.  Then
>> we'd need to figure what of the existing Java implementation might be
>> reused or adapted.  At a glance, it doesn't look to me like a few
>> one-line changes would suffice, adding methods where things are
>> hardwired, that rather more substantial changes would be required, but
>> I might be missing something.  If you're interested in pursuing this
>> then please file an issue in Jira where you can propose changes to the
>> specification and implementation.
>>
>> Cheers,
>>
>> Doug
>>
>> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <[EMAIL PROTECTED]>
>> wrote:
>> > I guess my question is more basic - given that this is somewhat
>> specific to
>> > my own use case:
>> >
>> > How does one use other forms of Encoder/Decoder implementations that are
>> > available in the Avro library along with the Avro-Ipc SDK.
>> >
>> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc
>> supports is
>> > the BinaryEncoding
>> >
>> > Pankaj
>> >
>> >
>> >
>> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> Doug
>> >>
>> >> Perhaps you answered a portion of my conundrum in another thread
>> >> (permalink below) - but there is still the handshake and reuse of
>> invocation
>> >> logic question. Let me also think about this a little bit.
>> >>
>> >> Thanks in any case. Avro is a great tool in any case!
>> >>
>> >>
>> >>
>> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
>> >>
>> >>
>> >> Pankaj
>> >>
>> >> [EMAIL PROTECTED]
>> >>
>> >>
>> >>
>> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <[EMAIL PROTECTED]>
>> wrote:
>> >>>
>> >>> This sounds like a different RPC wire format than Avro's.  Avro's
>> >>> Requestor and Responder implement Avro's RPC wire format.  Avro's
>> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
>> >>> implementation of other RPC wire formats that include Avro data.
>> >>> Avro's Transceiver API may or may not be reusable, since it assumes
>> >>> Avro-style framing.  Parts of Requestor and Responder *might* be
>> >>> reusable and some refactoring of those classes *might* make such reuse
>> >>> easier, but there's not that much logic there that's not specific to
>> >>> Avro's wire format, so it might be just as easy to reimplement this
>> >>> layer for a different wire format.  It's hard for me to say without
>> >>> seeing a patch with a proposed refactoring.  Does that make sense?
>> >>>
>> >>> Doug
>> >>
Pankaj Shroff
[EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB