Hello,

We have been using  Avro successfully to serialize many of our objects,
using binary encoding, for storage and retrieval.  Although the
documentation about the Reflect Mapping states:
     This API is not recommended except as a stepping stone for systems
that currently uses Java interfaces to define RPC protocols.
we used this mapping as that recommendation did not seem to apply.  We
do not use the serialized data for RPC (or any other messaging
system).    In fact, this part has in-place for a while and works
exceptionally well.

Now we would like to "externalize" a smaller subset of the objects for
interaction with a WebApp.  Here we would like to use the JSON encoding
and the "specific" mapping.    We tried having this set of objects
implement "GenericRecord", however, this then breaks the use of
Reflection on these objects.  [The ReflectData.createSchema method
checks for this condition.]

Can Avro be used to serialize objects one way, and externalize them
another?  [The externalized objects are a subset of the serialized
ones.]   Perhaps more generally, my question is: can both binary
encoding and JSON encoding be supported on overlapping objects using
different mappers?   If yes, what is the best way to accomplish this?

Thanks for any help - I am still quite a noob here so I greatly
appreciate any additional details!

Joe Gamache

On 4/23/12 10:37 AM, "Joe Gamache" <[EMAIL PROTECTED]> wrote:

> Hello,
>
> We have been using  Avro successfully to serialize many of our objects, using
> binary encoding, for storage and retrieval.  Although the documentation about
> the Reflect Mapping states:
>     This API is not recommended except as a stepping stone for systems that
> currently uses Java interfaces to define RPC protocols.
> we used this mapping as that recommendation did not seem to apply.  We do not
> use the serialized data for RPC (or any other messaging system).    In fact,
> this part has in-place for a while and works exceptionally well.
>
> Now we would like to "externalize" a smaller subset of the objects for
> interaction with a WebApp.  Here we would like to use the JSON encoding and
> the "specific" mapping.    We tried having this set of objects implement
> "GenericRecord", however, this then breaks the use of Reflection on these
> objects.  [The ReflectData.createSchema method checks for this condition.]
>
> Can Avro be used to serialize objects one way, and externalize them another?
> [The externalized objects are a subset of the serialized ones.]   Perhaps more
> generally, my question is: can both binary encoding and JSON encoding be
> supported on overlapping objects using different mappers?   If yes, what is
> the best way to accomplish this?

That should be possible.  If not I think It is a bug.  The Java reflect API
is supposed to be able to handle Specific and Generic records, or at least
there is supposed to be a way to use them both.
What is the specific error, from what API call?  Perhaps it is a simple fix
and you can submit a patch and test to JIRA?

Thanks,

-Scott

>
> Thanks for any help - I am still quite a noob here so I greatly appreciate any
> additional details!
>
> Joe Gamache

Scott,

Thanks for the reply.  I agree it should be possible, but these lines
from ReflectData, createSchema method:

         } else if (GenericFixed.class.isAssignableFrom(c)) { // fixed
           int size = c.getAnnotation(FixedSize.class).value();
           schema = Schema.createFixed(name, null /* doc */, space, size);
         } else if (IndexedRecord.class.isAssignableFrom(c)) { // specific
           return super.createSchema(type, names);

execute INSTEAD of falling through to the else clause right afterwards
(which would be the desired behavior).  Sorry, but I can't remember
which clause executed, it was one of the "isAssignableFrom"s that came
back true.

Having said all this, we have decided just to use jackson directly.  It
looks like that is what avro uses internally, as well as what Spring
uses (also out of the box).

Joe

On 5/1/2012 7:30 PM, Scott Carey wrote:
>
>
> On 4/23/12 10:37 AM, "Joe Gamache" <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Hello,
>
>     We have been using  Avro successfully to serialize many of our
>     objects, using binary encoding, for storage and retrieval.
>     Although the documentation about the Reflect Mapping states:
>         This API is not recommended except as a stepping stone for
>     systems that currently uses Java interfaces to define RPC protocols.
>     we used this mapping as that recommendation did not seem to
>     apply.  We do not use the serialized data for RPC (or any other
>     messaging system).    In fact, this part has in-place for a while
>     and works exceptionally well.
>
>     Now we would like to "externalize" a smaller subset of the objects
>     for interaction with a WebApp.  Here we would like to use the JSON
>     encoding and the "specific" mapping.    We tried having this set
>     of objects implement "GenericRecord", however, this then breaks
>     the use of Reflection on these objects.  [The
>     ReflectData.createSchema method checks for this condition.]
>
>     Can Avro be used to serialize objects one way, and externalize
>     them another?  [The externalized objects are a subset of the
>     serialized ones.]   Perhaps more generally, my question is: can
>     both binary encoding and JSON encoding be supported on overlapping
>     objects using different mappers?   If yes, what is the best way to
>     accomplish this?
>
>
> That should be possible.  If not I think It is a bug.  The Java
> reflect API is supposed to be able to handle Specific and Generic
> records, or at least there is supposed to be a way to use them both.
> What is the specific error, from what API call?  Perhaps it is a
> simple fix and you can submit a patch and test to JIRA?
>
> Thanks,
>
> -Scott
>
>
>     Thanks for any help - I am still quite a noob here so I greatly
>     appreciate any additional details!
>
>     Joe Gamache
>