Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> MR job output report


Copy link to this message
-
Re: MR job output report
The logs of any job can be seen with following command:

hadoop job -history all output-directory-for-the-job.
for example: hadoop job -history all /user/hduser3/sort-output

On Mon, Mar 11, 2013 at 3:32 PM, John Meza <[EMAIL PROTECTED]> wrote:

> All,
> Q. How does everyone refer to the typical output report of a MapReduce job
> (below)? What is it called?
> Q. Isn't that "report" saved somewhere? Is it reproduciable? I've looked
> at the HDFS job
>  output directory which contains a log, but that doesn't have all the
> info as the report.
> Q. How can I redirect the output of the distcp (next line) to a logfile.
> I've tried alot of the
>     typical command line redirects(>,>>,&>,2>..). They didn't work.:
> hadoop distcp -i hdfs://had2100.zim.com:8020/akamai/2013/02/01
>               hdfs://zip4.zim.com/akamai/2013/02/$n
>
> many thanks,
> John
> -----Typical MR output report------
> 13/03/11 13:35:53 INFO mapred.LocalJobRunner: Copied: 495 Skipped: 0
> Failed: 0
> 13/03/11 13:35:53 INFO mapred.Task: Task
> 'attempt_local1065909426_0001_m_000000_0' done.
> 13/03/11 13:35:53 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local1065909426_0001_m_000000_0
> 13/03/11 13:35:53 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/03/11 13:35:53 INFO mapred.JobClient:  map 100% reduce 0%
> 13/03/11 13:35:53 INFO mapred.JobClient: Job complete:
> job_local1065909426_0001
> 13/03/11 13:35:53 INFO mapred.JobClient: Counters: 22
> 13/03/11 13:35:53 INFO mapred.JobClient:   File System Counters
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of bytes
> read=525640
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of bytes
> written=518996
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of read
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of large read
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     FILE: Number of write
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of bytes
> read=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of bytes
> written=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of read
> operations=3026
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of large read
> operations=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     HDFS: Number of write
> operations=1514
> 13/03/11 13:35:53 INFO mapred.JobClient:   Map-Reduce Framework
> 13/03/11 13:35:53 INFO mapred.JobClient:     Map input records=519
> 13/03/11 13:35:53 INFO mapred.JobClient:     Map output records=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     Input split bytes=145
> 13/03/11 13:35:53 INFO mapred.JobClient:     Spilled Records=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     CPU time spent (ms)=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     Physical memory (bytes)
> snapshot=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     Virtual memory (bytes)
> snapshot=0
> 13/03/11 13:35:53 INFO mapred.JobClient:     Total committed heap usage
> (bytes)=409468928
> 13/03/11 13:35:53 INFO mapred.JobClient:
> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
> 13/03/11 13:35:53 INFO mapred.JobClient:     BYTES_READ=102066
> 13/03/11 13:35:53 INFO mapred.JobClient:   distcp
> 13/03/11 13:35:53 INFO mapred.JobClient:     Bytes copied=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient:     Bytes expected=27505977242
> 13/03/11 13:35:53 INFO mapred.JobClient:     Files copied=495
>
>
--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member

http://homepage.usask.ca/~jac735/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB