Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Technical question on Capacity Scheduler.


Copy link to this message
-
Re: Technical question on Capacity Scheduler.
I am observing the logs.

On Tue, Mar 5, 2013 at 9:02 PM, Jagmohan Chauhan <[EMAIL PROTECTED]
> wrote:

> I think i need to re-frame the question a bit. Yes CS supports multiple
> jobs in parallel but of different users. If i am submitting 3 different
> jobs for a same user at the same time, they  are always executed in a FIFO
> manner.
> Am i correct?
>
> On Tue, Mar 5, 2013 at 8:55 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> The CS does support running jobs in parallel. Are you observing just
>> the UI or are also noticing a FIFO behavior in logs where assignments
>> can be seen with timestamps?
>>
>> On Wed, Mar 6, 2013 at 9:03 AM, Jagmohan Chauhan
>> <simplefunduare [EMAIL PROTECTED] <[EMAIL PROTECTED]>> wrote:
>> > Hi All
>> >
>> > Can someone please reply to my queries?
>> >
>> > On Sun, Mar 3, 2013 at 5:47 PM, Jagmohan Chauhan <
>> [EMAIL PROTECTED]
>> >> wrote:
>> >
>> >> Thanks Harsh.
>> >>
>> >> I have a few more questions.
>> >>
>> >> Q1: I found it in my experiments using CS that for any user , its next
>> job
>> >> does not start until its current one is finished. Is it true and are
>> there
>> >> any exceptions and if true then why is it so?  I I did not find any
>> such
>> >> condition in the implementation of CS.
>> >>
>> >> Q2: The concept of reserved slots  is true only if speculative
>> execution
>> >> is on. Am i correct ? If yes,then the code dealing with reserved slots
>> wont
>> >> be executed if speculative execution is off?
>> >>
>> >> PS: I am working on MRv1.
>> >>
>> >>
>> >> On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>> >>
>> >>> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
>> >>> [EMAIL PROTECTED]
>> >>> > wrote:
>> >>>
>> >>> >  Hi
>> >>> >
>> >>> > I am going through the Capacity Scheduler implementation. There is
>> one
>> >>> > thing i did not understand clearly.
>> >>> >
>> >>>
>> >>> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
>> >>> suggest reading the newer one for any implementation or research
>> goals,
>> >>> for
>> >>> it to be more current and future-applicable.
>> >>>
>> >>>
>> >>> > 1. Does the o ff-switch task refers to a task in which data has to
>> be
>> >>> > fetched over the network. It means its not node-local ?
>> >>> >
>> >>>
>> >>> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
>> >>>
>> >>>
>> >>> > 2. Does off-switch task  includes only the tasks for which map input
>> >>> has to
>> >>> > be fetched from a node on a different rack across the switch or it
>> also
>> >>> > includes task where data has to be fetched from another node on same
>> >>> rack
>> >>> > on same switch?
>> >>> >
>> >>>
>> >>> A task's input split is generally supposed to define all locations of
>> >>> available inputs. If the CS is unable to schedule to any of those
>> >>> locations, nor their racks, then it schedules an off-rack (see above)
>> task
>> >>> which has to pull the input from a different rack.
>> >>>
>> >>>
>> >>> >
>> >>> > --
>> >>> > Thanks and Regards
>> >>> > Jagmohan Chauhan
>> >>> > MSc student,CS
>> >>> > Univ. of Saskatchewan
>> >>> > IEEE Graduate Student Member
>> >>> >
>> >>> > http://homepage.usask.ca/~jac735/
>> >>> >
>> >>>
>> >>> Feel free to post any further impl. related questions! :)
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Thanks and Regards
>> >> Jagmohan Chauhan
>> >> MSc student,CS
>> >> Univ. of Saskatchewan
>> >> IEEE Graduate Student Member
>> >>
>> >> http://homepage.usask.ca/~jac735/
>> >>
>> >
>> >
>> >
>> > --
>> > Thanks and Regards
>> > Jagmohan Chauhan
>> > MSc student,CS
>> > Univ. of Saskatchewan
>> > IEEE Graduate Student Member
>> >
>> > http://homepage.usask.ca/~jac735/
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
>

--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member

http://homepage.usask.ca/~jac735/