Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Can dynamic Classes be accessible to Mappers/Reducers?


Copy link to this message
-
Can dynamic Classes be accessible to Mappers/Reducers?
Jay Vyas 2012-11-12, 23:58
Hi guys:

Im trying to dynamically create a java class at runtime and submit it as a
hadoop job.

How does the Mapper (or for that matter, Reducer) use the data in the Job
object?  That is, how does it load a class?  Is the job object serialized,
along with all the info necessary to load a class?

The reason im wondering is that, in all reality, the class im creating will
not be on the classpath of JVM's in a distributed environment.  But indeed,
it will exist when the Job is created .  So Im wondering wether simply
"creating"  a dynamic class in side of the job executioner will be
serialized and sent over the wire in such a way that it can be instantiated
in a different JVM or not.

--
Jay Vyas
http://jayunit100.blogspot.com
+
Zizon Qiu 2012-11-13, 01:54
+
Bertrand Dechoux 2012-11-13, 08:55
+
Jay Vyas 2012-11-13, 11:50