Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Collecting union-ed Records in AvroReducer


+
Andrew Kenworthy 2011-12-08, 12:10
Copy link to this message
-
Re: Collecting union-ed Records in AvroReducer
Gaurav Nanda 2011-12-08, 14:32
You don't need to construct a record object. You can just write your
RecordA/RecorbB objects directly.

Sample Writer:
            DatumWriter<Object> datum = new GenericDatumWriter<Object>(schema);
   DataFileWriter<Object> writer = new DataFileWriter<Object>(datum);

        FileOutputStream out = new FileOutputStream("h:\\TestFile.avro");

   writer.create(schema, out);
   writer.append(1050324); //You can write your recordA/recordB here.

   writer.close();

Sample Reader:

            File out = new File("h:\\TestFile.avro");
            GenericDatumReader<Object> datum = new GenericDatumReader<Object>();
   DataFileReader<Object> reader = new DataFileReader<Object>(out, datum);

            while (reader.hasNext()) {
     System.out.println(reader.next());
   }
   reader.close();

Hope this helps.

Thanks,
Gaurav Nanda

On Thu, Dec 8, 2011 at 5:40 PM, Andrew Kenworthy <[EMAIL PROTECTED]> wrote:
> Hallo,
>
> is it possible to write/collect a union-ed record from an avro reducer?
>
> I have a reduce class (extending AvroReducer), and the output schema is a
> union schema of record type A and record type B. In the reduce logic I want
> to combine instances of A and B in the same datum, passing it to my
> Avrocollector. My code looks a bit like this:
>
> Record unionRecord = new GenericData.Record(myUnionSchema); // not legal!
> unionRecord.put("type A", recordA);
> unionRecord.put("type B", recordB);
> collector.collect(unionRecord);
>
> but GenericData.Record constructor expects a Record Schema. How can I write
> both records such that they appear in the same output datum?
>
> Andrew
+
Doug Cutting 2011-12-08, 17:05
+
Scott Carey 2011-12-08, 17:45
+
Andrew Kenworthy 2011-12-13, 10:27
+
Andrew Kenworthy 2011-12-08, 15:03