Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Submitting and running hadoop jobs Programmatically

Copy link to this message
Re: Submitting and running hadoop jobs Programmatically
 I am using the same APIs but i am not able to run the jobs by just adding
the configuration files and jars . It never create a job in Hadoop , it just
shows cleaning up staging area and fails.

On Tue, Jul 26, 2011 at 3:46 PM, Devaraj K <[EMAIL PROTECTED]> wrote:

> Hi Madhu,
>   You can submit the jobs using the Job API's programmatically from any
> system. The job submission code can be written this way.
>     // Create a new Job
>     Job job = new Job(new Configuration());
>     job.setJarByClass(MyJob.class);
>     // Specify various job-specific parameters
>     job.setJobName("myjob");
>     job.setInputPath(new Path("in"));
>     job.setOutputPath(new Path("out"));
>     job.setMapperClass(MyJob.MyMapper.class);
>     job.setReducerClass(MyJob.MyReducer.class);
>     // Submit the job
>     job.submit();
> For submitting this, need to add the hadoop jar files and configuration
> files in the class path of the application from where you want to submit
> the
> job.
> You can refer this docs for more info on Job API's.
> http://hadoop.apache.org/mapreduce/docs/current/api/org/apache/hadoop/mapred
> uce/Job.html
> Devaraj K
> -----Original Message-----
> From: madhu phatak [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, July 26, 2011 3:29 PM
> Subject: Submitting and running hadoop jobs Programmatically
> Hi,
>  I am working on a open source project
> Nectar<https://github.com/zinnia-phatak-dev/Nectar> where
> i am trying to create the hadoop jobs depending upon the user input. I was
> using Java Process API to run the bin/hadoop shell script to submit the
> jobs. But it seems not good way because the process creation model is
> not consistent across different operating systems . Is there any better way
> to submit the jobs rather than invoking the shell script? I am using
> hadoop-0.21.0 version and i am running my program in the same user where
> hadoop is installed . Some of the older thread told if I add configuration
> files in path it will work fine . But i am not able to run in that way . So
> anyone tried this before? If So , please can you give detailed instruction
> how to achieve it . Advanced thanks for your help.
> Regards,
> Madhukara Phatak