Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Collecting union-ed Records in AvroReducer

Andrew Kenworthy 2011-12-08, 12:10
Copy link to this message
Re: Collecting union-ed Records in AvroReducer
You don't need to construct a record object. You can just write your
RecordA/RecorbB objects directly.

Sample Writer:
            DatumWriter<Object> datum = new GenericDatumWriter<Object>(schema);
   DataFileWriter<Object> writer = new DataFileWriter<Object>(datum);

        FileOutputStream out = new FileOutputStream("h:\\TestFile.avro");

   writer.create(schema, out);
   writer.append(1050324); //You can write your recordA/recordB here.


Sample Reader:

            File out = new File("h:\\TestFile.avro");
            GenericDatumReader<Object> datum = new GenericDatumReader<Object>();
   DataFileReader<Object> reader = new DataFileReader<Object>(out, datum);

            while (reader.hasNext()) {

Hope this helps.

Gaurav Nanda

On Thu, Dec 8, 2011 at 5:40 PM, Andrew Kenworthy <[EMAIL PROTECTED]> wrote:
> Hallo,
> is it possible to write/collect a union-ed record from an avro reducer?
> I have a reduce class (extending AvroReducer), and the output schema is a
> union schema of record type A and record type B. In the reduce logic I want
> to combine instances of A and B in the same datum, passing it to my
> Avrocollector. My code looks a bit like this:
> Record unionRecord = new GenericData.Record(myUnionSchema); // not legal!
> unionRecord.put("type A", recordA);
> unionRecord.put("type B", recordB);
> collector.collect(unionRecord);
> but GenericData.Record constructor expects a Record Schema. How can I write
> both records such that they appear in the same output datum?
> Andrew
Doug Cutting 2011-12-08, 17:05
Scott Carey 2011-12-08, 17:45
Andrew Kenworthy 2011-12-13, 10:27
Andrew Kenworthy 2011-12-08, 15:03