Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - Multiple resource requests for a given node (or all nodes)?


Copy link to this message
-
Re: Multiple resource requests for a given node (or all nodes)?
Todd Lipcon 2011-12-11, 00:21
On Sat, Dec 10, 2011 at 12:23 PM, Patrick Wendell
<[EMAIL PROTECTED]> wrote:

> What happens if an application wants to request multiple container
> types on a given node. E.g. say I need 10 2GB containers and 10 1GB
> containers, and I don't care which node they are on (i.e. RMNode.ANY).
> I really want to store 2 resource requests under RMNode.ANY in this
> case... don't I?
>
> Is the model just that an AM would ask for these in series?

My hunch is that this was overlooked because the resource sizes for MR
are basically set on a per-task-type level. That is, maps need X MB
and reduces need Y MB. Since maps and reduces are set at different
'priorities', they haven't conflicted.

Does it seem straightforward to change it to a multimap? Guava has a
nice implementation.

-Todd
--
Todd Lipcon
Software Engineer, Cloudera