Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Execute hadoop job remotely and programmatically


+
Yexi Jiang 2013-12-09, 20:58
Copy link to this message
-
Re: Execute hadoop job remotely and programmatically
Steve Lewis 2013-12-09, 22:00
out them in  a lib directory in the jar you pass to Hadoop and they will be
found
On Mon, Dec 9, 2013 at 12:58 PM, Yexi Jiang <[EMAIL PROTECTED]> wrote:

> Hi, All,
>
> I am working on a project that requires to execute a hadoop job remotely
> and the job requires some third-part libraries (jar files).
>
> Based on my understanding, I tried:
>
> 1. Copy these jar files to hdfs.
> 2. Copy them into the distributed cache using
> DistributedCache.addFileToClassPath so that hadoop can spread these jar
> files to each of the slave nodes.
>
> However, my program still throws ClassNotFoundException. Indicating that
> some of the classes cannot be found when the job is running.
>
> So I'm wondering:
> 1. What is the correct way to run a job remotely and programmatically
> while the job requires some third-party jar files.
> 2. I found DistributedCache is deprecated (I'm using hadoop 1.2.0), what
> is the alternative class?
>
> Regards,
> Yexi
>

--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com