Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Technical question on Capacity Scheduler.


Copy link to this message
-
Re: Technical question on Capacity Scheduler.
Thanks Harsh.

I have a few more questions.

Q1: I found it in my experiments using CS that for any user , its next job
does not start until its current one is finished. Is it true and are there
any exceptions and if true then why is it so?  I I did not find any such
condition in the implementation of CS.

Q2: The concept of reserved slots  is true only if speculative execution is
on. Am i correct ? If yes,then the code dealing with reserved slots wont be
executed if speculative execution is off?

PS: I am working on MRv1.
On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
> [EMAIL PROTECTED]
> > wrote:
>
> >  Hi
> >
> > I am going through the Capacity Scheduler implementation. There is one
> > thing i did not understand clearly.
> >
>
> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
> suggest reading the newer one for any implementation or research goals, for
> it to be more current and future-applicable.
>
>
> > 1. Does the o ff-switch task refers to a task in which data has to be
> > fetched over the network. It means its not node-local ?
> >
>
> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
>
>
> > 2. Does off-switch task  includes only the tasks for which map input has
> to
> > be fetched from a node on a different rack across the switch or it also
> > includes task where data has to be fetched from another node on same rack
> > on same switch?
> >
>
> A task's input split is generally supposed to define all locations of
> available inputs. If the CS is unable to schedule to any of those
> locations, nor their racks, then it schedules an off-rack (see above) task
> which has to pull the input from a different rack.
>
>
> >
> > --
> > Thanks and Regards
> > Jagmohan Chauhan
> > MSc student,CS
> > Univ. of Saskatchewan
> > IEEE Graduate Student Member
> >
> > http://homepage.usask.ca/~jac735/
> >
>
> Feel free to post any further impl. related questions! :)
>
> --
> Harsh J
>

--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member

http://homepage.usask.ca/~jac735/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB