|
|
-
hadoop queueLin Ma 2013-01-28, 03:02
Hi guys,
It is mentioned below for Hadoop queue ( http://blog.cloudera.com/blog/2008/11/job-scheduling-in-hadoop/), for the comments "FIFO scheduling with priorities is used", does it mean for resources (mapper/reducer slots) allocation inside each queue, two factors ((1) job submit order, i.e. FIFO, and (2) individual job priorities are also considered)? So it means even if inside one queue, the resource allocation is not fair between jobs? *However, within each queue, FIFO scheduling with priorities is used, except for one aspect – you can place a limit on percent of running tasks per user, so that users share a cluster equally. In other words, the capacity scheduler tries to simulate a separate FIFO/priority cluster for each user and each organization, rather than performing fair sharing between all jobs. * regards, Lin |