Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> resolving schemas in multiple avro data files


Copy link to this message
-
resolving schemas in multiple avro data files
We've recently come across a situation where we have two data files with
different schemas that we'd like to process together using
GenericDatumReader.  One schema is promotable to the other, but not vice
versa.  We'd like to programmatically determine which of the schemas to use.
 I did a brief look through javadoc and tests, and I couldn't find any
examples of checking if one schema is promotable to the other.  Has anyone
else come across this?
For some context, we're considering patching AvroStorage [1] to remove the
assumption that all files have the same schema.  In our case, our schema has
evolved in that a field that was an int was promoted to a long.
Thanks,

Joe

[1] https://issues.apache.org/jira/browse/PIG-1748
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB