Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - execute hadoop job from remote web application


Copy link to this message
-
Re: execute hadoop job from remote web application
Oleg Ruchovets 2011-10-18, 12:59
Thanks  you all for your answers but I still have a questions:
  Currently we running our jobs using shell scripts which locates on hadoop
master machine.

Here is an example of command line:
/opt/hadoop/bin/hadoop jar /opt/hadoop/hadoop-jobs/my_hadoop_job.jar
-inputPath /opt/inputs/  -outputPath /data/output_jobs/output

my_hadoop_job.jar has a class which parse input parameters and submit a job.
Our code is very similar like you wrote:
   ......

        job.setJarByClass(HadoopJobExecutor.class);
        job.setMapperClass(MultipleOutputMap.class);
        job.setCombinerClass(BaseCombine.class);
        job.setReducerClass(HBaseReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(MapWritable.class);

        FileOutputFormat.setOutputPath(job, new Path(finalOutPutPath));

        jobCompleteStatus = job.waitForCompletion(true);
...............

my question are:

1) my_hadoop_job.jar contains another classes (business logic) not only
Map,Combine,Reduce classes and I still don't understand how can I submit job
which needs all classes from my_hadoop_job.jar?
2) Do I need to submit a my_hadoop_job.jar too? If yes what is the way to do
it?

Thanks In Advance
Oleg.

On Tue, Oct 18, 2011 at 2:11 PM, Uma Maheswara Rao G 72686 <
[EMAIL PROTECTED]> wrote:

> ----- Original Message -----
> From: Bejoy KS <[EMAIL PROTECTED]>
> Date: Tuesday, October 18, 2011 5:25 pm
> Subject: Re: execute hadoop job from remote web application
> To: [EMAIL PROTECTED]
>
> > Oleg
> >      If you are looking at how to submit your jobs using
> > JobClient then the
> > below sample can give you a start.
> >
> > //get the configuration parameters and assigns a job name
> >        JobConf conf = new JobConf(getConf(), MyClass.class);
> >        conf.setJobName("SMS Reports");
> >
> >        //setting key value types for mapper and reducer outputs
> >        conf.setOutputKeyClass(Text.class);
> >        conf.setOutputValueClass(Text.class);
> >
> >        //specifying the custom reducer class
> >        conf.setReducerClass(SmsReducer.class);
> >
> >        //Specifying the input directories(@ runtime) and Mappers
> > independently for inputs from multiple sources
> >        FileInputFormat.addInputPath(conf, new Path(args[0]));
> >
> >        //Specifying the output directory @ runtime
> >        FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> >
> >        JobClient.runJob(conf);
> >
> > Along with the hadoop jars you may need to have the config files
> > as well on
> > your client.
> >
> > The sample is from old map reduce API. You can use the new one as
> > well in
> > that we use the Job instead of JobClient.
> >
> > Hope it helps!..
> >
> > Regards
> > Bejoy.K.S
> >
> >
> > On Tue, Oct 18, 2011 at 5:00 PM, Oleg Ruchovets
> > <[EMAIL PROTECTED]>wrote:
> > > Excellent. Can you give a small example of code.
> > >
> Good samle by Bejoy
> hope, you have access for this site.
> Also please go through this docs,
>
> http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#Example%3A+WordCount+v2.0
> Here is the wordcount example.
>
> > >
> > > On Tue, Oct 18, 2011 at 1:13 PM, Uma Maheswara Rao G 72686 <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > >
> > > > ----- Original Message -----
> > > > From: Oleg Ruchovets <[EMAIL PROTECTED]>
> > > > Date: Tuesday, October 18, 2011 4:11 pm
> > > > Subject: execute hadoop job from remote web application
> > > > To: [EMAIL PROTECTED]
> > > >
> > > > > Hi , what is the way to execute hadoop job on remote
> > cluster. I
> > > > > want to
> > > > > execute my hadoop job from remote web  application , but I
> > didn't> > > find any
> > > > > hadoop client (remote API) to do it.
> > > > >
> > > > > Please advice.
> > > > > Oleg
> > > > >
> > > > You can put the Hadoop jars in your web applications classpath
> > and find
> > > the
> > > > Class JobClient and submit the jobs using it.
> > > >
> > > > Regards,
> > > > Uma
> > > >
>