Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Submitting and running hadoop jobs Programmatically


Copy link to this message
-
Re: Submitting and running hadoop jobs Programmatically
I am using JobControl.add() to add a job and running job control in
a separate thread and using JobControl.allFinished() to see all jobs
completed or not . Is this work same as Job.submit()??

On Tue, Jul 26, 2011 at 4:08 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Madhu,
>
> Do you get a specific error message / stack trace? Could you also
> paste your JT logs?
>
> On Tue, Jul 26, 2011 at 4:05 PM, madhu phatak <[EMAIL PROTECTED]>
> wrote:
> > Hi
> >  I am using the same APIs but i am not able to run the jobs by just
> adding
> > the configuration files and jars . It never create a job in Hadoop , it
> just
> > shows cleaning up staging area and fails.
> >
> > On Tue, Jul 26, 2011 at 3:46 PM, Devaraj K <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Madhu,
> >>
> >>   You can submit the jobs using the Job API's programmatically from any
> >> system. The job submission code can be written this way.
> >>
> >>     // Create a new Job
> >>     Job job = new Job(new Configuration());
> >>     job.setJarByClass(MyJob.class);
> >>
> >>     // Specify various job-specific parameters
> >>     job.setJobName("myjob");
> >>
> >>     job.setInputPath(new Path("in"));
> >>     job.setOutputPath(new Path("out"));
> >>
> >>     job.setMapperClass(MyJob.MyMapper.class);
> >>     job.setReducerClass(MyJob.MyReducer.class);
> >>
> >>     // Submit the job
> >>     job.submit();
> >>
> >>
> >>
> >> For submitting this, need to add the hadoop jar files and configuration
> >> files in the class path of the application from where you want to submit
> >> the
> >> job.
> >>
> >> You can refer this docs for more info on Job API's.
> >>
> >>
> http://hadoop.apache.org/mapreduce/docs/current/api/org/apache/hadoop/mapred
> >> uce/Job.html
> >>
> >>
> >>
> >> Devaraj K
> >>
> >> -----Original Message-----
> >> From: madhu phatak [mailto:[EMAIL PROTECTED]]
> >> Sent: Tuesday, July 26, 2011 3:29 PM
> >> To: [EMAIL PROTECTED]
> >> Subject: Submitting and running hadoop jobs Programmatically
> >>
> >> Hi,
> >>  I am working on a open source project
> >> Nectar<https://github.com/zinnia-phatak-dev/Nectar> where
> >> i am trying to create the hadoop jobs depending upon the user input. I
> was
> >> using Java Process API to run the bin/hadoop shell script to submit the
> >> jobs. But it seems not good way because the process creation model is
> >> not consistent across different operating systems . Is there any better
> way
> >> to submit the jobs rather than invoking the shell script? I am using
> >> hadoop-0.21.0 version and i am running my program in the same user where
> >> hadoop is installed . Some of the older thread told if I add
> configuration
> >> files in path it will work fine . But i am not able to run in that way .
> So
> >> anyone tried this before? If So , please can you give detailed
> instruction
> >> how to achieve it . Advanced thanks for your help.
> >>
> >> Regards,
> >> Madhukara Phatak
> >>
> >>
> >
>
>
>
> --
> Harsh J
>