Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> How can I limit reducers to one-per-node?


Copy link to this message
-
How can I limit reducers to one-per-node?
I have a cluster of boxes with 3 reducers per node. I want to limit a
particular job to only run 1 reducer per node.

 

This job is network IO bound, gathering images from a set of webservers.

 

My job has certain parameters set to meet "web politeness" standards (e.g.
limit connects and connection frequency).

 

If this job runs from multiple reducers on the same node, those per-host
limits will be violated.  Also, this is a shared environment and I don't
want long running network bound jobs uselessly taking up all reduce slots.

+
Nan Zhu 2013-02-09, 03:59
+
David Parks 2013-02-09, 04:24
+
Nan Zhu 2013-02-09, 04:30
+
David Parks 2013-02-09, 04:46
+
Nan Zhu 2013-02-09, 04:59
+
Harsh J 2013-02-09, 05:18