Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Scala API


Hi people,

Is there interest in a custom Scala API for Avro records and protocols?
I am currently working on an schema compiler for Scala, but before I go
deeper, I would really like to have external feedback.
I would especially like to hear from anyone who has opinions on how to map
Avro types onto Scala types.
Here are a few hints on what I've been trying so far:

   - Records are compiled into two forms: mutable and immutable.
   - To avoid collisions with Java generated classes, scala classes are
   generated in a .scala sub-package.
   - Avro arrays are translated to Seq/List when immutable and
   Buffer/ArrayBuffer when mutable.
   - Avro maps are translated to immutable or mutable Map/HashMap.
   - Bytes/Fixed are translated to Seq[Byte] when immutable and
   Buffer[Byte] when mutable.
   - Avro unions are currently translated into Any, but I plan to:
   - translate union{null, X} into Scala Option[X]
      - compile union {T1, T2, T3} into a custom case classes to have
      proper type checking and pattern matching.
   - Scala records provide a method encode(encoder) to serialize as binary
   into a byte stream (appears ~30% faster than SpecificDatumWriter).
   - Scala mutable records provide a method decode(decoder) to deserialize
   a byte stream (appears ~25% faster than SpecificDatumReader).
   - Scala records implement the SpecificRecord Java interface (with some
   overhead), so one may still use the SpecificDatumReader/Writer when the
   custom encoder/decoder methods cannot be used.
   - Mutable records can be converted to immutable (ie. can act as
   builders).

Thanks,
Christophe
+
Philip Zeyliger 2012-05-30, 16:34
+
Michael Armbrust 2012-06-05, 21:53
+
Scott Carey 2012-05-30, 21:52
+
Christophe Taton 2012-05-30, 23:26
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB