Marshall Bockrath-Vandegr... 2013-05-08, 22:33
-Re: Hadoop serialization DatumReader/Writer
Scott Carey 2013-05-13, 22:08
Making the DatumReader/Writers configurable would be a welcome addition.
Ideally, much more of what goes on there could be:
1. configuration driven
2. pre-computed to avoid repeated work during decoding/encoding
We do some of both already. The trick is to do #1 without impacting
performance and #2 requires a bigger overhaul.
If you would like, a contribution including a Clojure related maven module
or two that depends on the Java stuff would be a welcome addition and
allow us to identify compatibility issues as we change the Java library
On 5/8/13 3:33 PM, "Marshall Bockrath-Vandegrift" <[EMAIL PROTECTED]>
>Is there a reason Avro¹s Hadoop serialization classes don¹t allow
>configuration of the DatumReader and DatumWriter classes?
>My use-case is that I¹m implementing Clojure DatumReader and -Writer
>classes which produce and consume Clojure¹s data structures directly.
>I¹d like to then extend that to Hadoop MapReduce jobs which operate in
>terms of Clojure data, with Avro handling all de/serialization directly
>to/from that Clojure data.
>Am I going around this in a backwards fashion, or would a patch to allow
>configuration of the Hadoop serialization DatumReader/Writers be
Marshall Bockrath-Vandegr... 2013-05-13, 23:22