Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: each stage's time in hadoop


Copy link to this message
-
Re: each stage's time in hadoop
You can also try the following two commands:

1, hadoop job -status <job-id>
For example:
hadoop job -status job_201303021057_0004

I will get the following output:
Job: job_201303021057_0004
file:
hdfs://master:54310/user/ec2-user/.staging/job_201303021057_0004/job.xml
tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201303021057_0004
*map() completion: 0.8*
*reduce() completion: 0.0*

Counters: 18
        Job Counters
                SLOTS_MILLIS_MAPS=455464
                Launched map tasks=8
        File Input Format Counters
                Bytes Read=0
        File Output Format Counters
                Bytes Written=5239209962
        FileSystemCounters
...

2, hadoop job -events <job-id> <from-event-#> <#-of-events>
For example, if I want to check the first 5 events of a job starting from
index 0, the following command can be used:
hadoop -events job_201303021057_0004 0 5

We can get the following output, showing us the status of each job:
Task completion events for job_201303021057_0004
Number of events (from 0) are: 5
SUCCEEDED attempt_201303021057_0004_m_000006_0
http://slave4:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000006_0
SUCCEEDED attempt_201303021057_0004_m_000003_0
http://slave2:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000003_0
SUCCEEDED attempt_201303021057_0004_m_000000_0
http://slave4:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000000_0
SUCCEEDED attempt_201303021057_0004_m_000004_0
http://slave1:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000004_0
SUCCEEDED attempt_201303021057_0004_m_000002_0
http://slave5:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000002_0

Thanks,
Shumin

On Wed, Mar 6, 2013 at 4:18 AM, bharath vissapragada <
[EMAIL PROTECTED]> wrote:

> Look into JobHistory Class !
>
> On Wed, Mar 6, 2013 at 2:37 PM, Mirko Kämpf <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > please have a look on the "Starfish" project.
> >
> > http://www.cs.duke.edu/starfish/
> >
> > Best wishes
> > Mirko
> >
> >
> > 2013/3/6 claytonly <[EMAIL PROTECTED]>
> >>
> >> Hello ,all
> >>
> >>      I was using hadoop-1.0.0 in ubuntu 12.04. I was wondering how I can
> >> know each stage's running time in mapreduce. I got some information
> from web
> >> interface provided by hadoop, but I have to click each task one by one
> which
> >> is time-waste. How can I collect the information into one file ? Thank
> you
> >> very
> >> much !
> >>
> >> 2013-03-06
> >> ________________________________
> >> claytonly
> >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB