Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> Scheduler functionality in MRv2

Copy link to this message
Scheduler functionality in MRv2

Let's consider the following scenario

-> The MR Job has an InputSplit on host h1 and h2
-> AM makes a request to the Scheduler for a container on h1 and h2
-> The scheduler responds with containers c1 and c2 on h1 and h2
-> But the AM uses c1 and releases c2 after 15 minutes

In this scenario c2 cannot be allocated to any of the AM by the scheduler
for that span of 15 minutes and the c2 remains idle. And the released
container is also not accounted into the Queues quota. How can such
scenario be avoided?

Also, it would be nice to specify a more complex requests to the scheduler,
like a request for (h1 or h2), (r1 or r2). Add some dependencies between
the requests.