Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Configuring # task slots

Copy link to this message
Configuring # task slots
David Rosenstrauch 2010-08-19, 18:46
Was reading up a bit today on configuring the settings for # task slots,


Was just wondering:  couldn't (shouldn't?) this be done dynamically by
default?  i.e., couldn't/shouldn't a slave node be able to compute these
values programmatically based on the # of cores in the machine?
(Perhaps in conjunction with a mappers-to-reducers ratio, and a %
over-subscribed ratio.)

Obviously there'd be times where you'd want to manually override that,
but I'd think there could be a simple algorithm for computing this
(e.g., based on the info in slide #8 of this presentation:
that would cover most users' main use case.

Thoughts?  Is there something I'm overlooking here that would make this