Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Hadoop serialization DatumReader/Writer


Copy link to this message
-
Re: Hadoop serialization DatumReader/Writer
Making the DatumReader/Writers configurable would be a welcome addition.

Ideally, much more of what goes on there could be:
 1. configuration driven
 2. pre-computed to avoid repeated work during decoding/encoding

We do some of both already.  The trick is to do #1 without impacting
performance and #2 requires a bigger overhaul.

If you would like, a contribution including a Clojure related maven module
or two that depends on the Java stuff would be a welcome addition and
allow us to identify compatibility issues as we change the Java library
over time.
On 5/8/13 3:33 PM, "Marshall Bockrath-Vandegrift" <[EMAIL PROTECTED]>
wrote:

>Hi all:
>
>Is there a reason Avro¹s Hadoop serialization classes don¹t allow
>configuration of the DatumReader and DatumWriter classes?
>
>My use-case is that I¹m implementing Clojure DatumReader and -Writer
>classes which produce and consume Clojure¹s data structures directly.
>I¹d like to then extend that to Hadoop MapReduce jobs which operate in
>terms of Clojure data, with Avro handling all de/serialization directly
>to/from that Clojure data.
>
>Am I going around this in a backwards fashion, or would a patch to allow
>configuration of the Hadoop serialization DatumReader/Writers be
>welcome?
>
>-Marshall
>