Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> Scheduler functionality in MRv2


Copy link to this message
-
Scheduler functionality in MRv2
Hi,

Let's consider the following scenario

-> The MR Job has an InputSplit on host h1 and h2
-> AM makes a request to the Scheduler for a container on h1 and h2
-> The scheduler responds with containers c1 and c2 on h1 and h2
-> But the AM uses c1 and releases c2 after 15 minutes

In this scenario c2 cannot be allocated to any of the AM by the scheduler
for that span of 15 minutes and the c2 remains idle. And the released
container is also not accounted into the Queues quota. How can such
scenario be avoided?

Also, it would be nice to specify a more complex requests to the scheduler,
like a request for (h1 or h2), (r1 or r2). Add some dependencies between
the requests.

Thanks,
Praveen
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB