Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Referencing a Thrift sub-object within an Avro AVDL?


+
Ryan Slobojan 2012-08-15, 18:00
Copy link to this message
-
Re: Referencing a Thrift sub-object within an Avro AVDL?
On Wed, Aug 15, 2012 at 11:00 AM, Ryan Slobojan <[EMAIL PROTECTED]> wrote:
> I'm trying to figure out how to integrate some existing Thrift objects into
> an Avro-generated object, and haven't been able to find any pointers in the
> docs - as background, the project in question has recently adopted Avro as a
> new serialization standard, however there is still quite a bit of legacy
> code (and data) which uses Thrift, so the hope is that we can package a
> fairly small Thrift object in as a field on an AVDL-generated Avro object. I
> came across the org.apache.avro.thrift package
> (http://avro.apache.org/docs/1.7.1/api/java/org/apache/avro/thrift/package-summary.html)
> and see that reading and writing of Thrift objects is supported (as well as
> the test which shows it in action at
> http://svn.apache.org/viewvc/avro/tags/release-1.7.1/lang/java/thrift/src/test/java/org/apache/avro/thrift/TestThrift.java?view=markup),
> however it's unclear to me if (or how) I can point at a Thrift object within
> an AVDL.

That's not currently possible.

> Assuming that this AVDL embedding isn't possible (something tells me that
> ThriftDatum[Reader|Writer] being based off of GenericDatumReader means it's
> only meant for runtime use, and not as part of AVDL compilation), what would
> you recommend as the best approach towards achieving this? It would be
> possible to duplicate the Thrift object's schema in AVDL and create a second
> AVDL-based version of that object, but would there be a clean way to convert
> back and forth between the two representations without needing to add a
> bunch of extra code? The existing legacy code *really* wants a Thrift
> object, so I need to somehow get from the Avro object to that, preferably in
> the cleanest way possible - any pointers would be greatly appreciated.

You might alias the AVDL-based record names to the Thrift-based record
names, serialize the Thrift object to a buffer using ThriftDatumWriter
then deserialize it using SpecificDatumReader.  Would that work for
you?

Doug
+
Ryan Slobojan 2012-08-15, 19:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB