Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - MR job output report


+
John Meza 2013-03-11, 22:32
Copy link to this message
-
Re: MR job output report
Jagmohan Chauhan 2013-03-11, 22:36
The logs of any job can be seen with following command:

hadoop job -history all output-directory-for-the-job.
for example: hadoop job -history all /user/hduser3/sort-output

On Mon, Mar 11, 2013 at 3:32 PM, John Meza <[EMAIL PROTECTED]> wrote:

> All,
> Q. How does everyone refer to the typical output report of a MapReduce job
> (below)? What is it called?
> Q. Isn't that "report" saved somewhere? Is it reproduciable? I've looked
> at the HDFS job
>  output directory which contains a log, but that doesn't have all the
> info as the report.
> Q. How can I redirect the output of the distcp (next line) to a logfile.
> I've tried alot of the
>     typical command line redirects(>,>>,&>,2>..). They didn't work.:
> hadoop distcp -i hdfs://had2100.zim.com:8020/akamai/2013/02/01
>               hdfs://zip4.zim.com/akamai/2013/02/$n
>
> many thanks,
> John
> -----Typical MR output report------
> 13/03/11 13:35:53 INFO mapred.LocalJobRunner: Copied: 495 Skipped: 0
> Failed: 0
> 13/03/11 13:35:53 INFO mapred.Task: Task
> 'attempt_local1065909426_0001_m_000000_0' done.
> 13/03/11 13:35:53 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local1065909426_0001_m_000000_0
> 13/03/11 13:35:53 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/03/11 13:35:53 INFO mapred.JobClient:  map 100% reduce 0%
> 13/03/11 13:35:53 INFO mapred.JobClient: Job complete:
> job_local1065909426_0001
> 13/03/11 13:35:53 INFO mapred.JobClient: Counters: 22
> 13/03/11 13:35:53 INFO mapred.JobClient:   File System Counters
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of bytes
> read=525640
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of bytes
> written=518996
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of read
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of large read
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of write
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of bytes
> read=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of bytes
> written=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of read
> operations=3026
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of large read
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of write
> operations=1514
> 13/03/11 13:35:53 INFO mapred.JobClient:   Map-Reduce Framework
> 13/03/11 13:35:53 INFO mapred.JobClient:     Map input records=519
> 13/03/11 13:35:53 INFO mapred.JobClient:     Map output records=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     Input split bytes=145
> 13/03/11 13:35:53 INFO mapred.JobClient:     Spilled Records=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     CPU time spent (ms)=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     Physical memory (bytes)
> snapshot=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     Virtual memory (bytes)
> snapshot=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     Total committed heap usage
> (bytes)=409468928
> 13/03/11 13:35:53 INFO mapred.JobClient:
> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
> 13/03/11 13:35:53 INFO mapred.JobClient:     BYTES_READ=102066
> 13/03/11 13:35:53 INFO mapred.JobClient:   distcp
> 13/03/11 13:35:53 INFO mapred.JobClient:     Bytes copied=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient:     Bytes expected=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient:     Files copied=495
>
>
--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member

http://homepage.usask.ca/~jac735/