Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Submitting and running hadoop jobs Programmatically


Copy link to this message
-
Re: Submitting and running hadoop jobs Programmatically
I am using JobControl.add() to add a job and running job control in
a separate thread and using JobControl.allFinished() to see all jobs
completed or not . Is this work same as Job.submit()??

On Tue, Jul 26, 2011 at 4:08 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Madhu,
>
> Do you get a specific error message / stack trace? Could you also
> paste your JT logs?
>
> On Tue, Jul 26, 2011 at 4:05 PM, madhu phatak <[EMAIL PROTECTED]>
> wrote:
> > Hi
> >  I am using the same APIs but i am not able to run the jobs by just
> adding
> > the configuration files and jars . It never create a job in Hadoop , it
> just
> > shows cleaning up staging area and fails.
> >
> > On Tue, Jul 26, 2011 at 3:46 PM, Devaraj K <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Madhu,
> >>
> >>   You can submit the jobs using the Job API's programmatically from any
> >> system. The job submission code can be written this way.
> >>
> >>     // Create a new Job
> >>     Job job = new Job(new Configuration());
> >>     job.setJarByClass(MyJob.class);
> >>
> >>     // Specify various job-specific parameters
> >>     job.setJobName("myjob");
> >>
> >>     job.setInputPath(new Path("in"));
> >>     job.setOutputPath(new Path("out"));
> >>
> >>     job.setMapperClass(MyJob.MyMapper.class);
> >>     job.setReducerClass(MyJob.MyReducer.class);
> >>
> >>     // Submit the job
> >>     job.submit();
> >>
> >>
> >>
> >> For submitting this, need to add the hadoop jar files and configuration
> >> files in the class path of the application from where you want to submit
> >> the
> >> job.
> >>
> >> You can refer this docs for more info on Job API's.
> >>
> >>
> http://hadoop.apache.org/mapreduce/docs/current/api/org/apache/hadoop/mapred
> >> uce/Job.html
> >>
> >>
> >>
> >> Devaraj K
> >>
> >> -----Original Message-----
> >> From: madhu phatak [mailto:[EMAIL PROTECTED]]
> >> Sent: Tuesday, July 26, 2011 3:29 PM
> >> To: [EMAIL PROTECTED]
> >> Subject: Submitting and running hadoop jobs Programmatically
> >>
> >> Hi,
> >>  I am working on a open source project
> >> Nectar<https://github.com/zinnia-phatak-dev/Nectar> where
> >> i am trying to create the hadoop jobs depending upon the user input. I
> was
> >> using Java Process API to run the bin/hadoop shell script to submit the
> >> jobs. But it seems not good way because the process creation model is
> >> not consistent across different operating systems . Is there any better
> way
> >> to submit the jobs rather than invoking the shell script? I am using
> >> hadoop-0.21.0 version and i am running my program in the same user where
> >> hadoop is installed . Some of the older thread told if I add
> configuration
> >> files in path it will work fine . But i am not able to run in that way .
> So
> >> anyone tried this before? If So , please can you give detailed
> instruction
> >> how to achieve it . Advanced thanks for your help.
> >>
> >> Regards,
> >> Madhukara Phatak
> >>
> >>
> >
>
>
>
> --
> Harsh J
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB