Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - speculative execution in yarn


Copy link to this message
-
Re: speculative execution in yarn
Vinod Kumar Vavilapalli 2012-10-17, 18:28
Speculative execution is a per-job concept, so in 2.* release line, it is MR AM's responsibility.

Because there is a two level scheduling - one at RM and one at AM, AMs have not way of figuring out if there are other jobs are not. In general, AMs send container requests in a greedy manner and leave the responsibility of cross-job fairness to the RM and the scheduler.

In sum, to answer your question, speculative execution will be triggered only based on task run times, and irrespective of other jobs.

Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Oct 17, 2012, at 5:19 AM, Radim Kolar wrote:

> Is yarn smart enough to do speculative execution only if there are no other jobs running or queued?