Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Measuring Shuffle time for MR job


+
praveenesh kumar 2012-08-27, 06:18
+
Bertrand Dechoux 2012-08-27, 07:57
Copy link to this message
-
Re: Measuring Shuffle time for MR job
You can extract the shuffle time from the job log.

Take a look at 

https://github.com/rajvish/hadoop-summary 
Raj

>________________________________
> From: Bertrand Dechoux <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Sent: Monday, August 27, 2012 12:57 AM
>Subject: Re: Measuring Shuffle time for MR job
>
>Shuffle time is considered as part of the reduce step. Without reduce,
>there is no need for shuffling.
>One way to measure it would be using the full reduce time with a
>'/dev/null' reducer.
>
>I am not aware of any way to measure it.
>
>Regards
>
>Bertrand
>
>On Mon, Aug 27, 2012 at 8:18 AM, praveenesh kumar <[EMAIL PROTECTED]>wrote:
>
>> Is there a way to know the total shuffle time of a map-reduce job - I mean
>> some command or output  that can tell that ?
>>
>> I want to measure total map, total shuffle and total reduce time for my MR
>> job -- how can I achieve it ? I am using hadoop 0.20.205
>>
>>
>> Regards,
>> Praveenesh
>>
>
>
>
>--
>Bertrand Dechoux
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB