Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Union in AvroMapper.map: Not in Union??


Copy link to this message
-
Re: Union in AvroMapper.map: Not in Union??
On 10 April 2013 11:47, Doug Cutting <[EMAIL PROTECTED]> wrote:

> On Tue, Apr 9, 2013 at 8:55 PM, Martin Kleppmann <[EMAIL PROTECTED]>wrote:
>
>> AvroJob does not perform schema evolution, so the schema you provide
>> to AvroJob.setInputSchema has to be the exact same schema with which
>> your input files to the mappers are encoded.
>>
>
> Actually the schema you provide is used as the "reader" schema and need
> not exactly match the schema used to write the data.  The schema used when
> writing is stored in the data files.
>

Oh yes — sorry, I misread the code. (I overlooked the fact that
DataFileStream.initialize mutates the reader to use the writer's schema.)

Martin
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB