Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> what affects number of reducers launched by hadoop?


+
Vitaliy Semochkin 2010-07-28, 19:24
Copy link to this message
-
Re: what affects number of reducers launched by hadoop?
The 3 stages for reducer are:
copy
sort
reduce

On Wed, Jul 28, 2010 at 12:24 PM, Vitaliy Semochkin <[EMAIL PROTECTED]>wrote:

> Hi,
>
> in my cluster mapred.tasktracker.reduce.tasks.maximum = 4
> however during monitoring the job in job tracker I see only 1 reducer
> working
>
> first it is
> reduce > copy - can someone please explain what does this mean?
>
> after it is
> reduce > reduce
>
> when I set the number of reduce tasks for a job programatically to 10
> job.setNumReduceTasks(10);
> the number of "reduce > reduce" reducers increases to 10 and the
> performance of application increases as well (the number of reducers
> never exceeds).
>
> Can someone explain such behavior?
>
> Thanks in Advance,
> Vitaliy S
>
+
Joe Stein 2010-07-28, 19:40
+
Abhinay Mehta 2010-07-29, 10:31
+
Vitaliy Semochkin 2010-07-29, 11:20
+
Raj V 2010-07-29, 16:37
+
Joe Stein 2010-07-29, 12:54