Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Hadoop serialization DatumReader/Writer


Copy link to this message
-
Re: Hadoop serialization DatumReader/Writer
Making the DatumReader/Writers configurable would be a welcome addition.

Ideally, much more of what goes on there could be:
 1. configuration driven
 2. pre-computed to avoid repeated work during decoding/encoding

We do some of both already.  The trick is to do #1 without impacting
performance and #2 requires a bigger overhaul.

If you would like, a contribution including a Clojure related maven module
or two that depends on the Java stuff would be a welcome addition and
allow us to identify compatibility issues as we change the Java library
over time.
On 5/8/13 3:33 PM, "Marshall Bockrath-Vandegrift" <[EMAIL PROTECTED]>
wrote:

>Hi all:
>
>Is there a reason Avro¹s Hadoop serialization classes don¹t allow
>configuration of the DatumReader and DatumWriter classes?
>
>My use-case is that I¹m implementing Clojure DatumReader and -Writer
>classes which produce and consume Clojure¹s data structures directly.
>I¹d like to then extend that to Hadoop MapReduce jobs which operate in
>terms of Clojure data, with Avro handling all de/serialization directly
>to/from that Clojure data.
>
>Am I going around this in a backwards fashion, or would a patch to allow
>configuration of the Hadoop serialization DatumReader/Writers be
>welcome?
>
>-Marshall
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB