Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Deployment of jar files at startup

Copy link to this message
Re: Deployment of jar files at startup
On 09/14/2010 10:10 PM, Pete Tyler wrote:
> I'm trying to figure out how to achieve the following from a Java client,
> 1. My app (which is a web server) starts up
> 2. As part of startup my jar file, which includes my map reduce classes are distributed to hadoop nodes
> 3. My web app uses map reduce to extract data without the performance overhead of each job deploying a jar file, via setJar(), setJarByClass()
> It looks like DistributedCache() has potential but the need for commands like 'hadoop fs -copyFromLocal ...' and the API methods like '.getLocalCacheArchives()' look to be at odds with my scenario. Any thoughts?
> -Peter

For step 2, you have 2 options on how to implement:
a) call DistributedCache.addFileToClassPath(jarFileURI, conf);
b) have your app implement Tool, use ToolRunner to launch it, and
specify a -libjars command line parm which will achieve the same effect
as in (a).  See
for details.