Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: passing arguments to map reduce and queuing multiple jobs

Copy link to this message
Re: passing arguments to map reduce and queuing multiple jobs

This question is for mapreduce-user not hbase-user.

bcc hbase-user

On Wed, Feb 29, 2012 at 7:40 PM, T Vinod Gupta <[EMAIL PROTECTED]> wrote:
> hi,
> whats the recommended way of passing arguments to m/r jobs? based on web
> examples, the mapper and reducer classes are static classes. so if you have
> to set some parameters that the individual instances of m/r runs need to
> access, whats the way? im trying to do it by creating a static initialize
> method in mapper and reducer and configuring static variables there. i am
> not sure if this is the best way.

Look at the configure method:


You are passed the job conf. You can set your parameters on the job conf.

> this brings me to next question - i need to create a always running daemon
> that schedules multiple m/r jobs everyday. it can just queue multiple jobs
> and wait for their completion. but then the above question becomes even
> more relevant. if static variables are used, then how can one queue
> multiple jobs on the same mapper/reducer?

I think you want to look at oozie:  http://incubator.apache.org/oozie/


Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/