-Re: speculative execution in yarn
Vinod Kumar Vavilapalli 2012-10-17, 18:28
Speculative execution is a per-job concept, so in 2.* release line, it is MR AM's responsibility.
Because there is a two level scheduling - one at RM and one at AM, AMs have not way of figuring out if there are other jobs are not. In general, AMs send container requests in a greedy manner and leave the responsibility of cross-job fairness to the RM and the scheduler.
In sum, to answer your question, speculative execution will be triggered only based on task run times, and irrespective of other jobs.
+Vinod Kumar Vavilapalli
On Oct 17, 2012, at 5:19 AM, Radim Kolar wrote:
> Is yarn smart enough to do speculative execution only if there are no other jobs running or queued?