Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: Which Subphases Do Times on JobHistory Web UI Cover


Copy link to this message
-
Re: Which Subphases Do Times on JobHistory Web UI Cover
Average map time includes everything the map task is doing, i.e. all the
things you mentioned.  Reduce time does not cover shuffle time.  Reduce
time is the time spent calling the reducer function and writing its output
to HDFS.  Merge time is related to reduce, not map.

-Sandy
On Tue, Sep 24, 2013 at 6:57 PM, Efe Gencer <[EMAIL PROTECTED]> wrote:

> *By the way this question is about Apache Hadoop Release 2.1.0-beta.
>
> Thanks,
>
>
>
> 2013/9/24 Efe Gencer <[EMAIL PROTECTED]>
>
>> Hi All,
>>
>> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
>> ...
>> Started: <start time>
>> Finished: <finish time>
>> Elapsed: 12 mins, 5sec
>> Diagnostics:
>> *Average Map Time*: 1 mins, 40 sec
>> Average Reduce Time: 12 sec
>> Average Shuffle Time: 10 mins, 8 sec
>> Average Merge Time: 1 sec
>> ...
>>
>> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
>> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
>> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
>> to hdfs, etc)
>> 3) Is Average Merge time related with map or reduce? (since they both
>> have merge phases)
>>
>> Best,
>> Efe
>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB