Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Deployment of jar files at startup


Copy link to this message
-
Re: Deployment of jar files at startup
On 09/14/2010 10:10 PM, Pete Tyler wrote:
> I'm trying to figure out how to achieve the following from a Java client,
> 1. My app (which is a web server) starts up
> 2. As part of startup my jar file, which includes my map reduce classes are distributed to hadoop nodes
> 3. My web app uses map reduce to extract data without the performance overhead of each job deploying a jar file, via setJar(), setJarByClass()
>
> It looks like DistributedCache() has potential but the need for commands like 'hadoop fs -copyFromLocal ...' and the API methods like '.getLocalCacheArchives()' look to be at odds with my scenario. Any thoughts?
>
> -Peter

For step 2, you have 2 options on how to implement:
a) call DistributedCache.addFileToClassPath(jarFileURI, conf);
b) have your app implement Tool, use ToolRunner to launch it, and
specify a -libjars command line parm which will achieve the same effect
as in (a).  See
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/util/Tool.html
and
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/util/GenericOptionsParser.html#GenericOptions
for details.

HTH,

DR
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB