Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Can dynamic Classes be accessible to Mappers/Reducers?


Copy link to this message
-
Re: Can dynamic Classes be accessible to Mappers/Reducers?
when submiting a job,the ToolRunnuer or JobClient just distribute your jars
to hdfs,
so that tasktrackers can launch/"re-run" it.

In your case,you should have your dynamic class re-generate in
mapper/reducer`s setup method,
or the runtime classloader will miss them all.

On Tue, Nov 13, 2012 at 7:58 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> Hi guys:
>
> Im trying to dynamically create a java class at runtime and submit it as a
> hadoop job.
>
> How does the Mapper (or for that matter, Reducer) use the data in the Job
> object?  That is, how does it load a class?  Is the job object serialized,
> along with all the info necessary to load a class?
>
> The reason im wondering is that, in all reality, the class im creating
> will not be on the classpath of JVM's in a distributed environment.  But
> indeed, it will exist when the Job is created .  So Im wondering wether
> simply "creating"  a dynamic class in side of the job executioner will be
> serialized and sent over the wire in such a way that it can be instantiated
> in a different JVM or not.
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>