Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> Multiple resource requests for a given node (or all nodes)?


Copy link to this message
-
Re: Multiple resource requests for a given node (or all nodes)?
On Sat, Dec 10, 2011 at 12:23 PM, Patrick Wendell
<[EMAIL PROTECTED]> wrote:

> What happens if an application wants to request multiple container
> types on a given node. E.g. say I need 10 2GB containers and 10 1GB
> containers, and I don't care which node they are on (i.e. RMNode.ANY).
> I really want to store 2 resource requests under RMNode.ANY in this
> case... don't I?
>
> Is the model just that an AM would ask for these in series?

My hunch is that this was overlooked because the resource sizes for MR
are basically set on a per-task-type level. That is, maps need X MB
and reduces need Y MB. Since maps and reduces are set at different
'priorities', they haven't conflicted.

Does it seem straightforward to change it to a multimap? Guava has a
nice implementation.

-Todd
--
Todd Lipcon
Software Engineer, Cloudera
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB