Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Hadoop serialization DatumReader/Writer

Marshall Bockrath-Vandegr... 2013-05-08, 22:33
Copy link to this message
Re: Hadoop serialization DatumReader/Writer
Making the DatumReader/Writers configurable would be a welcome addition.

Ideally, much more of what goes on there could be:
 1. configuration driven
 2. pre-computed to avoid repeated work during decoding/encoding

We do some of both already.  The trick is to do #1 without impacting
performance and #2 requires a bigger overhaul.

If you would like, a contribution including a Clojure related maven module
or two that depends on the Java stuff would be a welcome addition and
allow us to identify compatibility issues as we change the Java library
over time.
On 5/8/13 3:33 PM, "Marshall Bockrath-Vandegrift" <[EMAIL PROTECTED]>

>Hi all:
>Is there a reason Avro¹s Hadoop serialization classes don¹t allow
>configuration of the DatumReader and DatumWriter classes?
>My use-case is that I¹m implementing Clojure DatumReader and -Writer
>classes which produce and consume Clojure¹s data structures directly.
>I¹d like to then extend that to Hadoop MapReduce jobs which operate in
>terms of Clojure data, with Avro handling all de/serialization directly
>to/from that Clojure data.
>Am I going around this in a backwards fashion, or would a patch to allow
>configuration of the Hadoop serialization DatumReader/Writers be
Marshall Bockrath-Vandegr... 2013-05-13, 23:22