Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> what affects number of reducers launched by hadoop?


+
Vitaliy Semochkin 2010-07-28, 19:24
Copy link to this message
-
Re: what affects number of reducers launched by hadoop?
The 3 stages for reducer are:
copy
sort
reduce

On Wed, Jul 28, 2010 at 12:24 PM, Vitaliy Semochkin <[EMAIL PROTECTED]>wrote:

> Hi,
>
> in my cluster mapred.tasktracker.reduce.tasks.maximum = 4
> however during monitoring the job in job tracker I see only 1 reducer
> working
>
> first it is
> reduce > copy - can someone please explain what does this mean?
>
> after it is
> reduce > reduce
>
> when I set the number of reduce tasks for a job programatically to 10
> job.setNumReduceTasks(10);
> the number of "reduce > reduce" reducers increases to 10 and the
> performance of application increases as well (the number of reducers
> never exceeds).
>
> Can someone explain such behavior?
>
> Thanks in Advance,
> Vitaliy S
>
+
Joe Stein 2010-07-28, 19:40
+
Abhinay Mehta 2010-07-29, 10:31
+
Vitaliy Semochkin 2010-07-29, 11:20
+
Raj V 2010-07-29, 16:37
+
Joe Stein 2010-07-29, 12:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB