Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - issue about total input byte of MR job


Copy link to this message
-
issue about total input byte of MR job
ch huang 2013-12-03, 08:28
i run the MR job,at the MR output i see

13/12/03 14:02:28 INFO mapreduce.JobSubmitter: number of splits:2717

because my each data block size is 64M,so total byte is 2717*64M/1024= 170G

but in the summary of end i see follow info ,so the HDFS read byte is
126792190158/1024/1024/1024 = 118G ,the two number is not very close ,why?

        File System Counters
                FILE: Number of bytes read=9642910241
                FILE: Number of bytes written=120327706125
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=126792190158
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=8151
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0