Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> speculative execution in yarn

Radim Kolar 2012-10-17, 12:19
Copy link to this message
Re: speculative execution in yarn
Speculative execution is a per-job concept, so in 2.* release line, it is MR AM's responsibility.

Because there is a two level scheduling - one at RM and one at AM, AMs have not way of figuring out if there are other jobs are not. In general, AMs send container requests in a greedy manner and leave the responsibility of cross-job fairness to the RM and the scheduler.

In sum, to answer your question, speculative execution will be triggered only based on task run times, and irrespective of other jobs.

+Vinod Kumar Vavilapalli
Hortonworks Inc.

On Oct 17, 2012, at 5:19 AM, Radim Kolar wrote:

> Is yarn smart enough to do speculative execution only if there are no other jobs running or queued?

Radim Kolar 2012-10-19, 17:00