Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Preferred way to submit a job?


Copy link to this message
-
Re: Preferred way to submit a job?
On 08/12/2010 12:10 AM, Harsh J wrote:
> On Thu, Aug 12, 2010 at 7:57 AM, David Rosenstrauch<[EMAIL PROTECTED]>  wrote:
>> On 08/11/2010 08:08 PM, Aaron Kimball wrote:
>>>> On a related note, if there's actually no difference between the 2
>>>> methods,
>>>> would anybody have any idea what could make the "mapred.job.tracker"
>>>> setting
>>>> on a job Configuration get ignored?  (I currently have it set to
>>>> "hdfs://<hadoop_job_tracker_host_name>:9001".)
>>>>
>>>>
>>> There's a reason that's being ignored :) That is not a jobtracker address.
>>> Assuming you've configured your namenode and your jobtracker on the same
>>> machine, then your fs.default.name should be hdfs://hdfs.host.name:port,
>>> and
>>> mapred.job.tracker should just be jt.host.name:port
>>>
>>> The port numbers in these two cases will be different.
>>
>> Hmmmm ... OK.  Not sure I understand why the syntax is different for thosee
>> 2 settings, but I'll give that a shot and see if it fixes the problem.
> Its probably because the JT has nothing to do with the HDFS protocols.
> Giving a hdfs:// scheme in its URI won't make sense :)

Actually, turns out that using an hdfs URI for the JT works just fine.
The problem turned out to be elsewhere.

(Gory details:  I was instantiating my Job with an empty configuration,
and then later setting properties on it (like the JT host/port).  Turns
out that Job decides *at instantiation time* whether the job will be run
locally or remotely, based on the contents of the JT property.  So my
setting the JT on the job's config after the fact was too late and had
no effect.)

Thanks,

DR
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB