Hadoop, mail # user - MR job output report

MR job output report

John Meza 2013-03-11, 22:32

All,Q. How does everyone refer to the typical output report of a MapReduce job (below)? What is it called?Q. Isn't that "report" saved somewhere? Is it reproduciable? I've looked at the HDFS job output directory which contains a log, but that doesn't have all the info as the report.Q. How can I redirect the output of the distcp (next line) to a logfile. I've tried alot of the typical command line redirects(>,>>,&>,2>..). They didn't work.:hadoop distcp -i hdfs://had2100.zim.com:8020/akamai/2013/02/01 hdfs://zip4.zim.com/akamai/2013/02/$n
many thanks,John-----Typical MR output report------13/03/11 13:35:53 INFO mapred.LocalJobRunner: Copied: 495 Skipped: 0 Failed: 013/03/11 13:35:53 INFO mapred.Task: Task 'attempt_local1065909426_0001_m_000000_0' done.13/03/11 13:35:53 INFO mapred.LocalJobRunner: Finishing task: attempt_local1065909426_0001_m_000000_013/03/11 13:35:53 INFO mapred.LocalJobRunner: Map task executor complete.13/03/11 13:35:53 INFO mapred.JobClient: map 100% reduce 0%13/03/11 13:35:53 INFO mapred.JobClient: Job complete: job_local1065909426_000113/03/11 13:35:53 INFO mapred.JobClient: Counters: 2213/03/11 13:35:53 INFO mapred.JobClient: File System Counters13/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of bytes read=52564013/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of bytes written=51899613/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of read operations=013/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of large read operations=013/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of write operations=013/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of bytes read=2750597724213/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of bytes written=2750597724213/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of read operations=302613/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of large read operations=013/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of write operations=151413/03/11 13:35:53 INFO mapred.JobClient: Map-Reduce Framework13/03/11 13:35:53 INFO mapred.JobClient: Map input records=51913/03/11 13:35:53 INFO mapred.JobClient: Map output records=013/03/11 13:35:53 INFO mapred.JobClient: Input split bytes=14513/03/11 13:35:53 INFO mapred.JobClient: Spilled Records=013/03/11 13:35:53 INFO mapred.JobClient: CPU time spent (ms)=013/03/11 13:35:53 INFO mapred.JobClient: Physical memory (bytes) snapshot=013/03/11 13:35:53 INFO mapred.JobClient: Virtual memory (bytes) snapshot=013/03/11 13:35:53 INFO mapred.JobClient: Total committed heap usage (bytes)=40946892813/03/11 13:35:53 INFO mapred.JobClient: org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter13/03/11 13:35:53 INFO mapred.JobClient: BYTES_READ=10206613/03/11 13:35:53 INFO mapred.JobClient: distcp13/03/11 13:35:53 INFO mapred.JobClient: Bytes copied=2750597724213/03/11 13:35:53 INFO mapred.JobClient: Bytes expected=2750597724213/03/11 13:35:53 INFO mapred.JobClient: Files copied=495

Re: MR job output report

Jagmohan Chauhan 2013-03-11, 22:36

The logs of any job can be seen with following command:

hadoop job -history all output-directory-for-the-job.
for example: hadoop job -history all /user/hduser3/sort-output

On Mon, Mar 11, 2013 at 3:32 PM, John Meza <[EMAIL PROTECTED]> wrote:

> All,
> Q. How does everyone refer to the typical output report of a MapReduce job
> (below)? What is it called?
> Q. Isn't that "report" saved somewhere? Is it reproduciable? I've looked
> at the HDFS job
> output directory which contains a log, but that doesn't have all the
> info as the report.
> Q. How can I redirect the output of the distcp (next line) to a logfile.
> I've tried alot of the
> typical command line redirects(>,>>,&>,2>..). They didn't work.:
> hadoop distcp -i hdfs://had2100.zim.com:8020/akamai/2013/02/01
> hdfs://zip4.zim.com/akamai/2013/02/$n
>
> many thanks,
> John
> -----Typical MR output report------
> 13/03/11 13:35:53 INFO mapred.LocalJobRunner: Copied: 495 Skipped: 0
> Failed: 0
> 13/03/11 13:35:53 INFO mapred.Task: Task
> 'attempt_local1065909426_0001_m_000000_0' done.
> 13/03/11 13:35:53 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local1065909426_0001_m_000000_0
> 13/03/11 13:35:53 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/03/11 13:35:53 INFO mapred.JobClient: map 100% reduce 0%
> 13/03/11 13:35:53 INFO mapred.JobClient: Job complete:
> job_local1065909426_0001
> 13/03/11 13:35:53 INFO mapred.JobClient: Counters: 22
> 13/03/11 13:35:53 INFO mapred.JobClient: File System Counters
> 13/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of bytes
> read=525640
> 13/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of bytes
> written=518996
> 13/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of read
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of large read
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient: FILE: Number of write
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of bytes
> read=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of bytes
> written=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of read
> operations=3026
> 13/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of large read
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient: HDFS: Number of write
> operations=1514
> 13/03/11 13:35:53 INFO mapred.JobClient: Map-Reduce Framework
> 13/03/11 13:35:53 INFO mapred.JobClient: Map input records=519
> 13/03/11 13:35:53 INFO mapred.JobClient: Map output records=0
> 13/03/11 13:35:53 INFO mapred.JobClient: Input split bytes=145
> 13/03/11 13:35:53 INFO mapred.JobClient: Spilled Records=0
> 13/03/11 13:35:53 INFO mapred.JobClient: CPU time spent (ms)=0
> 13/03/11 13:35:53 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=0
> 13/03/11 13:35:53 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=0
> 13/03/11 13:35:53 INFO mapred.JobClient: Total committed heap usage
> (bytes)=409468928
> 13/03/11 13:35:53 INFO mapred.JobClient:
> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
> 13/03/11 13:35:53 INFO mapred.JobClient: BYTES_READ=102066
> 13/03/11 13:35:53 INFO mapred.JobClient: distcp
> 13/03/11 13:35:53 INFO mapred.JobClient: Bytes copied=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient: Bytes expected=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient: Files copied=495
>
>
--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member

http://homepage.usask.ca/~jac735/