Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Re: Which Subphases Do Times on JobHistory Web UI Cover


Copy link to this message
-
Re: Which Subphases Do Times on JobHistory Web UI Cover
Sandy Ryza 2013-09-25, 02:17
Average map time includes everything the map task is doing, i.e. all the
things you mentioned.  Reduce time does not cover shuffle time.  Reduce
time is the time spent calling the reducer function and writing its output
to HDFS.  Merge time is related to reduce, not map.

-Sandy
On Tue, Sep 24, 2013 at 6:57 PM, Efe Gencer <[EMAIL PROTECTED]> wrote:

> *By the way this question is about Apache Hadoop Release 2.1.0-beta.
>
> Thanks,
>
>
>
> 2013/9/24 Efe Gencer <[EMAIL PROTECTED]>
>
>> Hi All,
>>
>> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
>> ...
>> Started: <start time>
>> Finished: <finish time>
>> Elapsed: 12 mins, 5sec
>> Diagnostics:
>> *Average Map Time*: 1 mins, 40 sec
>> Average Reduce Time: 12 sec
>> Average Shuffle Time: 10 mins, 8 sec
>> Average Merge Time: 1 sec
>> ...
>>
>> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
>> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
>> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
>> to hdfs, etc)
>> 3) Is Average Merge time related with map or reduce? (since they both
>> have merge phases)
>>
>> Best,
>> Efe
>>
>
>