Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Can dynamic Classes be accessible to Mappers/Reducers?


Copy link to this message
-
Can dynamic Classes be accessible to Mappers/Reducers?
Hi guys:

Im trying to dynamically create a java class at runtime and submit it as a
hadoop job.

How does the Mapper (or for that matter, Reducer) use the data in the Job
object?  That is, how does it load a class?  Is the job object serialized,
along with all the info necessary to load a class?

The reason im wondering is that, in all reality, the class im creating will
not be on the classpath of JVM's in a distributed environment.  But indeed,
it will exist when the Job is created .  So Im wondering wether simply
"creating"  a dynamic class in side of the job executioner will be
serialized and sent over the wire in such a way that it can be instantiated
in a different JVM or not.

--
Jay Vyas
http://jayunit100.blogspot.com
+
Zizon Qiu 2012-11-13, 01:54
+
Bertrand Dechoux 2012-11-13, 08:55
+
Jay Vyas 2012-11-13, 11:50
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB