Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> How to direct Reducer to write avro objects to avro sequence file?

Aleksey Maslov 2011-03-11, 05:54
Copy link to this message
Re: How to direct Reducer to write avro objects to avro sequence file?
By 'Avro sequence files' do you mean Avro data-files?

Avro-Mapred classes right now only support the older, stable API
(which has been undeprecated in 0.20.3, and is supported in 0.21 as
well - no worries in using it really). There is AVRO-593 that tracks a
new API implementation of Avro's mapred suppor (but it should be
fairly easy to write your own wrappers for these after a bit of
reading, since changes are mostly superficial).

On Fri, Mar 11, 2011 at 11:24 AM, Aleksey Maslov
> Hi,
> (using hadoop 0.20.2 and avro 1.4.1)
> I have defined a simple avro object 'AvroObj' (a record of strings),
> compiled the schema and
> setup a simple MR job that takes as input <Object, Text> and emits
> <Text, IntWritable>
> and reducer that takes said <Text, IntWritable> and ...
> I would like to achieve is - have reducer emit <NullWritable, AvroObj>
> pairs into an avro sequence file;
> so the next mr job will open that avro file and read-in avro objects, not
> text lines, out of it;
> I have looked through the (H ed.2) book and few online samples but can't
> figure out how to do it;
> some online sources mention job config settings like:
>        job.setOutputFormatClass(AvroOutputFormat.class);
>        AvroOutputFormat.setCompressOutput(conf, false);
> But this doesn't compile - setCompressOutput asks for deprecated JobConf
> object, and
> "setOutputFormatClass" gives error about its param - param not applicable to
> AvroOutputFormat.class;
> Could someone enlighten me how to have reducer write to avro sequence file ?
> Cheers;
> --
> View this message in context: http://apache-avro.679487.n3.nabble.com/How-to-direct-Reducer-to-write-avro-objects-to-avro-sequence-file-tp2663706p2663706.html
> Sent from the Avro - Users mailing list archive at Nabble.com.

Harsh J
Aleksey Maslov 2011-03-14, 20:54
Scott Carey 2011-03-14, 21:13