Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Measuring Shuffle time for MR job


Copy link to this message
-
Re: Measuring Shuffle time for MR job
You can extract the shuffle time from the job log.

Take a look at 

https://github.com/rajvish/hadoop-summary 
Raj

>________________________________
> From: Bertrand Dechoux <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Sent: Monday, August 27, 2012 12:57 AM
>Subject: Re: Measuring Shuffle time for MR job
>
>Shuffle time is considered as part of the reduce step. Without reduce,
>there is no need for shuffling.
>One way to measure it would be using the full reduce time with a
>'/dev/null' reducer.
>
>I am not aware of any way to measure it.
>
>Regards
>
>Bertrand
>
>On Mon, Aug 27, 2012 at 8:18 AM, praveenesh kumar <[EMAIL PROTECTED]>wrote:
>
>> Is there a way to know the total shuffle time of a map-reduce job - I mean
>> some command or output  that can tell that ?
>>
>> I want to measure total map, total shuffle and total reduce time for my MR
>> job -- how can I achieve it ? I am using hadoop 0.20.205
>>
>>
>> Regards,
>> Praveenesh
>>
>
>
>
>--
>Bertrand Dechoux
>
>
>