Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Avro MapReduce with Oozie


Copy link to this message
-
Re: Avro MapReduce with Oozie
Frank,

A simple hack would be this:

Write up a simple driver class as you would for regular, non-Oozie MR
code, configure your avro job as normal, and then instead of having
the submit API bits at the end, try:

jobConf.writeXml(outputStream);

This will give you a simple job config XML dump with the configs avro
has put in, and you can now port them into your <map-reduce> action as
config elements/etc.. Should be fairly trivial to port.

Will this work for you?

Please do share a template MR avro job workflow on the list if you've
got one in the end, in spirit of http://xkcd.com/979/ :)

(Hat tip to Alejandro from Oozie for the tip)

On Fri, Jan 27, 2012 at 1:06 AM, Frank Grimes <[EMAIL PROTECTED]> wrote:
> Hi All,
>
> We're trying to evaluate using Oozie (http://incubator.apache.org/oozie/) to
> run Hadoop MapReduce jobs over Avro data.
>
> As far as I can tell, Oozie configures the JobConf it submits to Hadoop
> based on external config files.
> see e.g.
>  http://mail-archives.apache.org/mod_mbox/avro-user/201110.mbox/<[EMAIL PROTECTED]>
>
> I'm wondering if anybody out there has an example of how to setup/run an
> Avro MapReduce job without relying on the AvroJob.set* helper methods.
> Or better yet, an Oozie example of the same.
>
> Thanks,
>
> Frank Grimes
>
>

--
Harsh J
Customer Ops. Engineer, Cloudera
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB