Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Problems adding JARs to distributed classpath in Hadoop 0.20.2


Copy link to this message
-
Re: Problems adding JARs to distributed classpath in Hadoop 0.20.2
John,

Now I get what you are trying to do.

My recommendation would be:

* Use a Java action to do all the stuff prior to starting your MR job
* Use a mapreduce action to start your MR job
* If you need to propagate properties from the Java action to the MR action
you can use the <capture-output> flag.

If you still want to start your MR job from your Java action, then your Java
action should do all the setup the MapReduceMain class does before starting
the MR job (this will ensure delegation tokens and distributed cache is
avail to your MR job).

Thanks.

Alejandro

On Mon, May 30, 2011 at 6:34 AM, John Armstrong <[EMAIL PROTECTED]>wrote:

> On Fri, 27 May 2011 15:47:23 -0700, Alejandro Abdelnur <[EMAIL PROTECTED]>
> wrote:
> > John,
> >
> > If you are using Oozie, dropping all the JARs your MR jobs needs in the
> > Oozie WF lib/ directory should suffice. Oozie will make sure all those
> JARs
> > are in the distributed cache.
>
> That doesn't seem to work.  I have this JAR in the WF /lib/ directory
> because the Java job that launches the MR job needs it.  And yes, it's in
> the distributed cache for the wrapper MR job that Oozie uses to remotely
> run the Java job.  The problem is it's not available for the MR job that
> the Java job launches.
>
> Thanks, though, for the suggestion.
>