Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Collecting union-ed Records in AvroReducer


+
Andrew Kenworthy 2011-12-08, 12:10
Copy link to this message
-
Re: Collecting union-ed Records in AvroReducer
You don't need to construct a record object. You can just write your
RecordA/RecorbB objects directly.

Sample Writer:
            DatumWriter<Object> datum = new GenericDatumWriter<Object>(schema);
   DataFileWriter<Object> writer = new DataFileWriter<Object>(datum);

        FileOutputStream out = new FileOutputStream("h:\\TestFile.avro");

   writer.create(schema, out);
   writer.append(1050324); //You can write your recordA/recordB here.

   writer.close();

Sample Reader:

            File out = new File("h:\\TestFile.avro");
            GenericDatumReader<Object> datum = new GenericDatumReader<Object>();
   DataFileReader<Object> reader = new DataFileReader<Object>(out, datum);

            while (reader.hasNext()) {
     System.out.println(reader.next());
   }
   reader.close();

Hope this helps.

Thanks,
Gaurav Nanda

On Thu, Dec 8, 2011 at 5:40 PM, Andrew Kenworthy <[EMAIL PROTECTED]> wrote:
> Hallo,
>
> is it possible to write/collect a union-ed record from an avro reducer?
>
> I have a reduce class (extending AvroReducer), and the output schema is a
> union schema of record type A and record type B. In the reduce logic I want
> to combine instances of A and B in the same datum, passing it to my
> Avrocollector. My code looks a bit like this:
>
> Record unionRecord = new GenericData.Record(myUnionSchema); // not legal!
> unionRecord.put("type A", recordA);
> unionRecord.put("type B", recordB);
> collector.collect(unionRecord);
>
> but GenericData.Record constructor expects a Record Schema. How can I write
> both records such that they appear in the same output datum?
>
> Andrew
+
Doug Cutting 2011-12-08, 17:05
+
Scott Carey 2011-12-08, 17:45
+
Andrew Kenworthy 2011-12-13, 10:27
+
Andrew Kenworthy 2011-12-08, 15:03
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB