Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Question about Hadoop Default FCFS Job Scheduler


Copy link to this message
-
Re: Question about Hadoop Default FCFS Job Scheduler
Nan Zhu 2011-01-17, 14:28
Hi, Chen

How is it going recently?

Actually I think you misundertand the code in assignTasks() in
JobQueueTaskScheduler.java, see the following structure of the interesting
codes:

//I'm sorry, I hacked the code so much, the name of the variables may be
different from the original version

for (i = 0; i < MapperCapacity; ++i){
   ...
   for (JobInProgress job:jobQueue){
       //try to shedule a node-local or rack-local map tasks
       //here is the interesting place
       t = job.obtainNewLocalMapTask(...);
       if (t != null){
          ...
          break;//the break statement here will make the control flow back
to "for (job:jobQueue)" which means that it will restart map tasks selection
procedure from the first job, so , it is actually schedule all of the first
job's local mappers first until the map slots are full
       }
   }
}

BTW, we can only schedule a reduce task in a single heartbeat

Best,
Nan
On Sat, Jan 15, 2011 at 1:45 PM, He Chen <[EMAIL PROTECTED]> wrote:

> Hey all
>
> Why does the FCFS scheduler only let a node chooses one task at a time in
> one job? In order to increase the data locality,
> it is reasonable to let a node to choose all its local tasks (if it can)
> from a job at a time.
>
> Any reply will be appreciated.
>
> Thanks
>
> Chen
>