Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Execute hadoop job remotely and programmatically


Copy link to this message
-
Re: Execute hadoop job remotely and programmatically
out them in  a lib directory in the jar you pass to Hadoop and they will be
found
On Mon, Dec 9, 2013 at 12:58 PM, Yexi Jiang <[EMAIL PROTECTED]> wrote:

> Hi, All,
>
> I am working on a project that requires to execute a hadoop job remotely
> and the job requires some third-part libraries (jar files).
>
> Based on my understanding, I tried:
>
> 1. Copy these jar files to hdfs.
> 2. Copy them into the distributed cache using
> DistributedCache.addFileToClassPath so that hadoop can spread these jar
> files to each of the slave nodes.
>
> However, my program still throws ClassNotFoundException. Indicating that
> some of the classes cannot be found when the job is running.
>
> So I'm wondering:
> 1. What is the correct way to run a job remotely and programmatically
> while the job requires some third-party jar files.
> 2. I found DistributedCache is deprecated (I'm using hadoop 1.2.0), what
> is the alternative class?
>
> Regards,
> Yexi
>

--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB