Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Can dynamic Classes be accessible to Mappers/Reducers?


Copy link to this message
-
Re: Can dynamic Classes be accessible to Mappers/Reducers?
when submiting a job,the ToolRunnuer or JobClient just distribute your jars
to hdfs,
so that tasktrackers can launch/"re-run" it.

In your case,you should have your dynamic class re-generate in
mapper/reducer`s setup method,
or the runtime classloader will miss them all.

On Tue, Nov 13, 2012 at 7:58 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> Hi guys:
>
> Im trying to dynamically create a java class at runtime and submit it as a
> hadoop job.
>
> How does the Mapper (or for that matter, Reducer) use the data in the Job
> object?  That is, how does it load a class?  Is the job object serialized,
> along with all the info necessary to load a class?
>
> The reason im wondering is that, in all reality, the class im creating
> will not be on the classpath of JVM's in a distributed environment.  But
> indeed, it will exist when the Job is created .  So Im wondering wether
> simply "creating"  a dynamic class in side of the job executioner will be
> serialized and sent over the wire in such a way that it can be instantiated
> in a different JVM or not.
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB