Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Preferred way to submit a job?


Copy link to this message
-
Re: Preferred way to submit a job?
On 08/11/2010 08:08 PM, Aaron Kimball wrote:
> On Wed, Aug 11, 2010 at 3:13 PM, David Rosenstrauch<[EMAIL PROTECTED]>wrote:
>
>> What's the preferred way to submit a job these days?
>> org.apache.hadoop.mapreduce.Job.submit() ?  Or
>> org.apache.hadoop.mapred.JobClient.runJob()?  Or does it even matter? (i.e.,
>> is there any difference between them?)
>>
>>
> If you're using the old API (e.g., you're filling out o.a.h.mapred.JobConf,
> and implementing o.a.h.mapred.Mapper) then you use JobClient.runJob(). If
> you're using the new API (o.a.h.mapreduce.Job, o.a.h.mapreduce.Mapper), then
> you use Job.waitForCompletion().
>
> You can't mix'n'match; your job has to be entirely "old style" or entirely
> "new style." Some programs use one, some use the other.

OK, so I'm not insane then.  :-)  That's how I thought it worked.
>> On a related note, if there's actually no difference between the 2 methods,
>> would anybody have any idea what could make the "mapred.job.tracker" setting
>> on a job Configuration get ignored?  (I currently have it set to
>> "hdfs://<hadoop_job_tracker_host_name>:9001".)
>>
>>
> There's a reason that's being ignored :) That is not a jobtracker address.
> Assuming you've configured your namenode and your jobtracker on the same
> machine, then your fs.default.name should be hdfs://hdfs.host.name:port, and
> mapred.job.tracker should just be jt.host.name:port
>
> The port numbers in these two cases will be different.

Hmmmm ... OK.  Not sure I understand why the syntax is different for
thosee 2 settings, but I'll give that a shot and see if it fixes the
problem.

Thanks much for the help!

DR