Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> How to direct Reducer to write avro objects to avro sequence file?


Copy link to this message
-
Re: How to direct Reducer to write avro objects to avro sequence file?
By 'Avro sequence files' do you mean Avro data-files?

Avro-Mapred classes right now only support the older, stable API
(which has been undeprecated in 0.20.3, and is supported in 0.21 as
well - no worries in using it really). There is AVRO-593 that tracks a
new API implementation of Avro's mapred suppor (but it should be
fairly easy to write your own wrappers for these after a bit of
reading, since changes are mostly superficial).

On Fri, Mar 11, 2011 at 11:24 AM, Aleksey Maslov
<[EMAIL PROTECTED]> wrote:
> Hi,
> (using hadoop 0.20.2 and avro 1.4.1)
>
> I have defined a simple avro object 'AvroObj' (a record of strings),
> compiled the schema and
> setup a simple MR job that takes as input <Object, Text> and emits
> <Text, IntWritable>
> and reducer that takes said <Text, IntWritable> and ...
> I would like to achieve is - have reducer emit <NullWritable, AvroObj>
> pairs into an avro sequence file;
>
> so the next mr job will open that avro file and read-in avro objects, not
> text lines, out of it;
>
> I have looked through the (H ed.2) book and few online samples but can't
> figure out how to do it;
> some online sources mention job config settings like:
>        job.setOutputFormatClass(AvroOutputFormat.class);
>        AvroOutputFormat.setCompressOutput(conf, false);
>
> But this doesn't compile - setCompressOutput asks for deprecated JobConf
> object, and
> "setOutputFormatClass" gives error about its param - param not applicable to
> AvroOutputFormat.class;
>
> Could someone enlighten me how to have reducer write to avro sequence file ?
>
> Cheers;
>
> --
> View this message in context: http://apache-avro.679487.n3.nabble.com/How-to-direct-Reducer-to-write-avro-objects-to-avro-sequence-file-tp2663706p2663706.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
>

--
Harsh J
www.harshj.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB