Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Deployment of jar files at startup


Copy link to this message
-
Re: Deployment of jar files at startup
On 09/14/2010 10:10 PM, Pete Tyler wrote:
> I'm trying to figure out how to achieve the following from a Java client,
> 1. My app (which is a web server) starts up
> 2. As part of startup my jar file, which includes my map reduce classes are distributed to hadoop nodes
> 3. My web app uses map reduce to extract data without the performance overhead of each job deploying a jar file, via setJar(), setJarByClass()
>
> It looks like DistributedCache() has potential but the need for commands like 'hadoop fs -copyFromLocal ...' and the API methods like '.getLocalCacheArchives()' look to be at odds with my scenario. Any thoughts?
>
> -Peter

For step 2, you have 2 options on how to implement:
a) call DistributedCache.addFileToClassPath(jarFileURI, conf);
b) have your app implement Tool, use ToolRunner to launch it, and
specify a -libjars command line parm which will achieve the same effect
as in (a).  See
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/util/Tool.html
and
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/util/GenericOptionsParser.html#GenericOptions
for details.

HTH,

DR