Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Re: each stage's time in hadoop


Copy link to this message
-
Re: each stage's time in hadoop
Shumin Guo 2013-03-06, 14:52
You can also try the following two commands:

1, hadoop job -status <job-id>
For example:
hadoop job -status job_201303021057_0004

I will get the following output:
Job: job_201303021057_0004
file:
hdfs://master:54310/user/ec2-user/.staging/job_201303021057_0004/job.xml
tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201303021057_0004
*map() completion: 0.8*
*reduce() completion: 0.0*

Counters: 18
        Job Counters
                SLOTS_MILLIS_MAPS=455464
                Launched map tasks=8
        File Input Format Counters
                Bytes Read=0
        File Output Format Counters
                Bytes Written=5239209962
        FileSystemCounters
...

2, hadoop job -events <job-id> <from-event-#> <#-of-events>
For example, if I want to check the first 5 events of a job starting from
index 0, the following command can be used:
hadoop -events job_201303021057_0004 0 5

We can get the following output, showing us the status of each job:
Task completion events for job_201303021057_0004
Number of events (from 0) are: 5
SUCCEEDED attempt_201303021057_0004_m_000006_0
http://slave4:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000006_0
SUCCEEDED attempt_201303021057_0004_m_000003_0
http://slave2:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000003_0
SUCCEEDED attempt_201303021057_0004_m_000000_0
http://slave4:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000000_0
SUCCEEDED attempt_201303021057_0004_m_000004_0
http://slave1:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000004_0
SUCCEEDED attempt_201303021057_0004_m_000002_0
http://slave5:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000002_0

Thanks,
Shumin

On Wed, Mar 6, 2013 at 4:18 AM, bharath vissapragada <
[EMAIL PROTECTED]> wrote:

> Look into JobHistory Class !
>
> On Wed, Mar 6, 2013 at 2:37 PM, Mirko Kämpf <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > please have a look on the "Starfish" project.
> >
> > http://www.cs.duke.edu/starfish/
> >
> > Best wishes
> > Mirko
> >
> >
> > 2013/3/6 claytonly <[EMAIL PROTECTED]>
> >>
> >> Hello ,all
> >>
> >>      I was using hadoop-1.0.0 in ubuntu 12.04. I was wondering how I can
> >> know each stage's running time in mapreduce. I got some information
> from web
> >> interface provided by hadoop, but I have to click each task one by one
> which
> >> is time-waste. How can I collect the information into one file ? Thank
> you
> >> very
> >> much !
> >>
> >> 2013-03-06
> >> ________________________________
> >> claytonly
> >
> >
>