Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - ReflectDatumReader and Writer classes


Copy link to this message
-
Re: ReflectDatumReader and Writer classes
Doug Cutting 2012-07-05, 18:18
Sounds like the word "simple" is confusing!  So instead how about we
just say, "See below for details and limitations"?

Doug

On Thu, Jul 5, 2012 at 11:01 AM, Peter Cameron
<[EMAIL PROTECTED]> wrote:
> That sounds fine to me. As long as "simple" is clearly defined by the
> "details and limitations".
>
> To us, such an object means a JavaBean with properties that are either
> primitive (where primitive maps to the Avro concept of primitive), or other
> JavaBeans. Since Avro is used for transmission over the wire, it's rather
> akin to a DTO. (We actually refer to these objects in code as "complex"
> because they are values that are not Avro primitives.)
>
> cheers,
> Peter
>
>
>
> On 05/07/2012 18:42, Doug Cutting wrote:
>>
>> Regardless of how we define "dynamic" that statement in the
>> documentation is confusing.  Folks do find Avro reflection useful in
>> many cases and we should improve that statement.
>>
>> Perhaps we should instead just say something like: "Transparently
>> supports simple classes.  See below for details and limitations."
>>
>> Would that be better?
>>
>> Thanks,
>>
>> Doug
>>
>> On Thu, Jul 5, 2012 at 10:18 AM, Peter Cameron
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> Let me explain further. Our data is not static. We do not know the type
>>> of
>>> Java object at runtime, as we only have the schema. We use the avro
>>> reflect
>>> package to transparently serialise and deserialise an Object instance
>>> given
>>> its schema. Ours is a black box that can serialise and deserialise any
>>> Object given a schema. We are given the Object to serialise by the
>>> caller,
>>> which is not under our control -- the only constraint is that both sides
>>> have the schema. The Specific readers/writers need code generation, and
>>> the
>>> generic readers and writers expect the objects to be "indexed records"
>>> and
>>> so barf. For any old POJO (with schema), the black box method can only be
>>> satisified by Avro's reflect package, unless I'm mistaken?
>>>
>>> Peter
>>>
>>>
>>>
>>> On 05/07/2012 18:09, Mark Hayes wrote:
>>>
>>> On Thu, Jul 5, 2012 at 9:44 AM, Peter Cameron
>>> <[EMAIL PROTECTED]>
>>> wrote:
>>>>
>>>> "This API is not recommended except as a stepping stone for systems that
>>>> currently uses Java interfaces to define RPC protocols. For new RPC
>>>> systems,
>>>> the specific API is preferred. For systems that process dynamic data,
>>>> the
>>>> generic API is probably best."
>>>>
>>>> What I'm confused by is the assertion that the generic API is "probably
>>>> best" for processing dynamic data.
>>>
>>>
>>> I am still fairly new to Avro but I think what the warning in the docs is
>>> trying to say is that the Specific API is better for static data, because
>>> reflection is slower.  If you're representing data using a Java bean,
>>> then
>>> your data is static (known at build time).
>>>
>>> --mark
>>>
>>>
>>>
>>> --
>>> Peter Cameron
>>> 2iC Limited
>>> T: +44 (0) 208 123 7479
>>> E: [EMAIL PROTECTED]
>>> W: www.2iCworld.com
>
>
>
> --
> Peter Cameron
> 2iC Limited
> T: +44 (0) 208 123 7479
> E: [EMAIL PROTECTED]
> W: www.2iCworld.com
>