Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> JobClient using deprecated JobConf


Copy link to this message
-
Re: JobClient using deprecated JobConf
On 09/25/2010 10:24 AM, Martin Becker wrote:
> Hello David,
>
> thanks a lot. Yet I want java code to submit my application. I do not
> want to mess with any kind of command line arguments or an executable,
> neither Java nor Hadoop. I want to write a method that can set up and
> submit a job to an arbitrary cluster. Something like calling
> CustomJob.submitJob(ip:port). This would be used by a GUI or another
> java application to process data.

Also easy to do.

All that the -jt and -fs parms do is eventually set appropriate values
in a Configuration object.  You can just as easily do this programmatically.

i.e., for -jt:

conf.set("mapred.job.tracker",
<hostname_of_job_tracker_in_remote_cluster:job_tracker_port_number>);

for -fs:
conf.set("fs.default.name",
<hdfs://hostname_of_name_node_in_remote_cluster:name_node_port_number>);
On a side note:  as far as your requirement "I do not want to mess with
any kind of command line arguments or an executable, neither Java nor
Hadoop", I'm not sure how feasible this requirement is because of the
-libjars command line parm.

It's easy to write code to handle what the -fs and -jt command line
parms do (see above).  But -libjars is much more complicated.  It takes
the list of jars that you give it, sends each one to the Hadoop
DistributedCache so that it gets distributed to each node in the
cluster, and then adds each one to the classpath of each map/reduce tasks.

Yes, I suppose you could try to write code to do that yourself, but I
can't see why you would want to reinvent the wheel here.  So this makes
me question whether that requirement really makes sense.
> The problem is the the
> missing job.jar, as also described earlier. I will start a new thread
> describing my problem using a more accurate header.
>
> Thank you,
> Martin

I'm not sure what you're referring to re: job.jar.

DR