Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Getting job information, joining a job in hadoop 0.20.1


Copy link to this message
-
Re: Getting job information, joining a job in hadoop 0.20.1
Saptarshi Guha 2010-05-26, 19:42
Hmm,
Oh well, a bit of effort pays off:
On 0.20.2

org.apache.hadoop.mapreduce.JobID jid org.apache.hadoop.mapreduce.JobID.forName(jd);
 org.apache.hadoop.mapred.JobClient jclient = new
org.apache.hadoop.mapred.JobClient(
    org.apache.hadoop.mapred.JobTracker.getAddress(new Configuration()),new
Configuration());
 org.apache.hadoop.mapred.JobID jj org.apache.hadoop.mapred.JobID.downgrade(jid);
org.apache.hadoop.mapred.RunningJob rj = jclient.getJob(jj);
 System.out.println(rj);
On Wed, May 26, 2010 at 2:48 PM, Saptarshi Guha <[EMAIL PROTECTED]>wrote:

> I found give, counters as obtained from RunningJob, I can still use it with
> the new style counters.
> But I'm still wondering how to get a RunningJob from a JobId.
> Is my Hadoop old?
>
> Regards
> Saptarshi
>
>
> On Wed, May 26, 2010 at 2:27 PM, Saptarshi Guha <[EMAIL PROTECTED]>wrote:
>
>> Hello,
>> I'm using Hadoop 0.20.1, I submitted a job using the
>> org.apache.hadoop.mapreduce.Job approach e.g.
>>
>> org.apache.hadoop.mapreduce.Job _job
>> job_.submit();
>>
>> However, I would like to,from another Java program, monitor this job:
>>
>> 1) get map/reduce progress
>> 2) get counters
>> 3) status
>> 4) and the ability to join i.e wait for completion.
>>
>> say jd = "job_201003190003_1855"
>>
>> org.apache.hadoop.mapreduce.JobID jid >> org.apache.hadoop.mapreduce.JobID.forName(jd);
>> org.apache.hadoop.mapred.JobClient jclient = new
>> org.apache.hadoop.mapred.JobClient();
>>
>> Now, what?
>>
>> http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapred/JobClient.html#getJob(org.apache.hadoop.mapred.JobID)
>>
>> says, I should use
>> 1. public RunningJob getJob(JobID jobid)
>>
>> but, in my installation of hadoop 0.20.1, i do not have such a method.
>>
>> 2. Even if I could, runningjob.getCounters() returns the old deprecated
>> Counters. Is this compatible with the new mapreduce code?
>>
>> Is there any org.apache.mapreduce  code that given the JobId, I can
>> monitor and wait for completion for the job?
>>
>> I can get a org.apache.hadoop.mapreduce.JobContext with the Job ID, but
>> there should be a way to get the Job.
>>
>> One option, is to serialize the job_ variable and load it in later(not my
>> first choice)
>> Thanks
>>
>> State: RUNNING
>> Started: Fri Mar 19 00:03:40 EDT 2010
>> Version: 0.20.1, r810220
>> Compiled: Tue Sep 1 20:55:56 UTC 2009 by oom
>> Identifier: 201003190003
>>
>>
>