Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Technical question on Capacity Scheduler.


Copy link to this message
-
Re: Technical question on Capacity Scheduler.
Thanks Harsh.

I have a few more questions.

Q1: I found it in my experiments using CS that for any user , its next job
does not start until its current one is finished. Is it true and are there
any exceptions and if true then why is it so?  I I did not find any such
condition in the implementation of CS.

Q2: The concept of reserved slots  is true only if speculative execution is
on. Am i correct ? If yes,then the code dealing with reserved slots wont be
executed if speculative execution is off?

PS: I am working on MRv1.
On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
> [EMAIL PROTECTED]
> > wrote:
>
> >  Hi
> >
> > I am going through the Capacity Scheduler implementation. There is one
> > thing i did not understand clearly.
> >
>
> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
> suggest reading the newer one for any implementation or research goals, for
> it to be more current and future-applicable.
>
>
> > 1. Does the o ff-switch task refers to a task in which data has to be
> > fetched over the network. It means its not node-local ?
> >
>
> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
>
>
> > 2. Does off-switch task  includes only the tasks for which map input has
> to
> > be fetched from a node on a different rack across the switch or it also
> > includes task where data has to be fetched from another node on same rack
> > on same switch?
> >
>
> A task's input split is generally supposed to define all locations of
> available inputs. If the CS is unable to schedule to any of those
> locations, nor their racks, then it schedules an off-rack (see above) task
> which has to pull the input from a different rack.
>
>
> >
> > --
> > Thanks and Regards
> > Jagmohan Chauhan
> > MSc student,CS
> > Univ. of Saskatchewan
> > IEEE Graduate Student Member
> >
> > http://homepage.usask.ca/~jac735/
> >
>
> Feel free to post any further impl. related questions! :)
>
> --
> Harsh J
>

--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member

http://homepage.usask.ca/~jac735/