Hadoop, mail # user - Re: Hadoop Streaming 0.20.2 and how to specify number of reducers per node -- is it possible?

Hadoop Streaming 0.20.2 and how to specify number of reducers per node -- is it possible?

Steven Hafran 2011-08-17, 07:36

hi everyone,

i have a few hadoop streaming tasks that would benefit from having the
reduce phase execute one reducer per node instead of two per node due to
high cpu and i/o. currently, i have a 30 node cluster and specify 30
reducers. when reviewing the job stats on the job tracker, i do see 30
reducers queued/executing; however, i have observed that those reducers are
distributed to 15 nodes resulting in only 50% use of my cluster.

after reviewing the hadoop docs, i've tried setting the following properties
when starting my streaming job; however, they don't seem to have any impact.
-jobconf mapred.tasktracker.reduce.tasks.maximum=1

how do i tell hadoop to run 1 reducer per node with streaming?

thanks in advance for your assistance!

regards,
-steven

Re: Hadoop Streaming 0.20.2 and how to specify number of reducers per node -- is it possible?

Allen Wittenauer 2011-08-22, 04:18

On Aug 17, 2011, at 12:36 AM, Steven Hafran wrote:
>
>
> after reviewing the hadoop docs, i've tried setting the following properties
> when starting my streaming job; however, they don't seem to have any impact.
> -jobconf mapred.tasktracker.reduce.tasks.maximum=1

"tasktracker" is the hint: that's a server side setting.

You're looking for the mapred.reduce.tasks settings.