Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> execute hadoop job from remote web application


+
Oleg Ruchovets 2011-10-18, 10:40
+
Steve Loughran 2011-10-18, 10:45
+
Uma Maheswara Rao G 72686... 2011-10-18, 11:13
+
Oleg Ruchovets 2011-10-18, 11:30
+
Devaraj K 2011-10-18, 11:51
+
Bejoy KS 2011-10-18, 11:54
+
Uma Maheswara Rao G 72686... 2011-10-18, 12:11
Copy link to this message
-
Re: execute hadoop job from remote web application
Thanks  you all for your answers but I still have a questions:
  Currently we running our jobs using shell scripts which locates on hadoop
master machine.

Here is an example of command line:
/opt/hadoop/bin/hadoop jar /opt/hadoop/hadoop-jobs/my_hadoop_job.jar
-inputPath /opt/inputs/  -outputPath /data/output_jobs/output

my_hadoop_job.jar has a class which parse input parameters and submit a job.
Our code is very similar like you wrote:
   ......

        job.setJarByClass(HadoopJobExecutor.class);
        job.setMapperClass(MultipleOutputMap.class);
        job.setCombinerClass(BaseCombine.class);
        job.setReducerClass(HBaseReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(MapWritable.class);

        FileOutputFormat.setOutputPath(job, new Path(finalOutPutPath));

        jobCompleteStatus = job.waitForCompletion(true);
...............

my question are:

1) my_hadoop_job.jar contains another classes (business logic) not only
Map,Combine,Reduce classes and I still don't understand how can I submit job
which needs all classes from my_hadoop_job.jar?
2) Do I need to submit a my_hadoop_job.jar too? If yes what is the way to do
it?

Thanks In Advance
Oleg.

On Tue, Oct 18, 2011 at 2:11 PM, Uma Maheswara Rao G 72686 <
[EMAIL PROTECTED]> wrote:

> ----- Original Message -----
> From: Bejoy KS <[EMAIL PROTECTED]>
> Date: Tuesday, October 18, 2011 5:25 pm
> Subject: Re: execute hadoop job from remote web application
> To: [EMAIL PROTECTED]
>
> > Oleg
> >      If you are looking at how to submit your jobs using
> > JobClient then the
> > below sample can give you a start.
> >
> > //get the configuration parameters and assigns a job name
> >        JobConf conf = new JobConf(getConf(), MyClass.class);
> >        conf.setJobName("SMS Reports");
> >
> >        //setting key value types for mapper and reducer outputs
> >        conf.setOutputKeyClass(Text.class);
> >        conf.setOutputValueClass(Text.class);
> >
> >        //specifying the custom reducer class
> >        conf.setReducerClass(SmsReducer.class);
> >
> >        //Specifying the input directories(@ runtime) and Mappers
> > independently for inputs from multiple sources
> >        FileInputFormat.addInputPath(conf, new Path(args[0]));
> >
> >        //Specifying the output directory @ runtime
> >        FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> >
> >        JobClient.runJob(conf);
> >
> > Along with the hadoop jars you may need to have the config files
> > as well on
> > your client.
> >
> > The sample is from old map reduce API. You can use the new one as
> > well in
> > that we use the Job instead of JobClient.
> >
> > Hope it helps!..
> >
> > Regards
> > Bejoy.K.S
> >
> >
> > On Tue, Oct 18, 2011 at 5:00 PM, Oleg Ruchovets
> > <[EMAIL PROTECTED]>wrote:
> > > Excellent. Can you give a small example of code.
> > >
> Good samle by Bejoy
> hope, you have access for this site.
> Also please go through this docs,
>
> http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#Example%3A+WordCount+v2.0
> Here is the wordcount example.
>
> > >
> > > On Tue, Oct 18, 2011 at 1:13 PM, Uma Maheswara Rao G 72686 <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > >
> > > > ----- Original Message -----
> > > > From: Oleg Ruchovets <[EMAIL PROTECTED]>
> > > > Date: Tuesday, October 18, 2011 4:11 pm
> > > > Subject: execute hadoop job from remote web application
> > > > To: [EMAIL PROTECTED]
> > > >
> > > > > Hi , what is the way to execute hadoop job on remote
> > cluster. I
> > > > > want to
> > > > > execute my hadoop job from remote web  application , but I
> > didn't> > > find any
> > > > > hadoop client (remote API) to do it.
> > > > >
> > > > > Please advice.
> > > > > Oleg
> > > > >
> > > > You can put the Hadoop jars in your web applications classpath
> > and find
> > > the
> > > > Class JobClient and submit the jobs using it.
> > > >
> > > > Regards,
> > > > Uma
> > > >
>
+
Bejoy KS 2011-10-18, 14:34
+
Oleg Ruchovets 2011-10-18, 15:43
+
Harsh J 2011-10-18, 16:13
+
Oleg Ruchovets 2011-10-18, 16:50
+
Harsh J 2011-10-18, 16:56
+
Steve Loughran 2011-10-20, 09:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB