|
|
-
Re: each stage's time in hadoopShumin Guo 2013-03-06, 14:52
You can also try the following two commands:
1, hadoop job -status <job-id> For example: hadoop job -status job_201303021057_0004 I will get the following output: Job: job_201303021057_0004 file: hdfs://master:54310/user/ec2-user/.staging/job_201303021057_0004/job.xml tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201303021057_0004 *map() completion: 0.8* *reduce() completion: 0.0* Counters: 18 Job Counters SLOTS_MILLIS_MAPS=455464 Launched map tasks=8 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=5239209962 FileSystemCounters ... 2, hadoop job -events <job-id> <from-event-#> <#-of-events> For example, if I want to check the first 5 events of a job starting from index 0, the following command can be used: hadoop -events job_201303021057_0004 0 5 We can get the following output, showing us the status of each job: Task completion events for job_201303021057_0004 Number of events (from 0) are: 5 SUCCEEDED attempt_201303021057_0004_m_000006_0 http://slave4:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000006_0 SUCCEEDED attempt_201303021057_0004_m_000003_0 http://slave2:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000003_0 SUCCEEDED attempt_201303021057_0004_m_000000_0 http://slave4:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000000_0 SUCCEEDED attempt_201303021057_0004_m_000004_0 http://slave1:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000004_0 SUCCEEDED attempt_201303021057_0004_m_000002_0 http://slave5:50060/tasklog?plaintext=true&attemptid=attempt_201303021057_0004_m_000002_0 Thanks, Shumin On Wed, Mar 6, 2013 at 4:18 AM, bharath vissapragada < [EMAIL PROTECTED]> wrote: > Look into JobHistory Class ! > > On Wed, Mar 6, 2013 at 2:37 PM, Mirko Kämpf <[EMAIL PROTECTED]> > wrote: > > Hi, > > > > please have a look on the "Starfish" project. > > > > http://www.cs.duke.edu/starfish/ > > > > Best wishes > > Mirko > > > > > > 2013/3/6 claytonly <[EMAIL PROTECTED]> > >> > >> Hello ,all > >> > >> I was using hadoop-1.0.0 in ubuntu 12.04. I was wondering how I can > >> know each stage's running time in mapreduce. I got some information > from web > >> interface provided by hadoop, but I have to click each task one by one > which > >> is time-waste. How can I collect the information into one file ? Thank > you > >> very > >> much ! > >> > >> 2013-03-06 > >> ________________________________ > >> claytonly > > > > > |