Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - question about when do resource matching in YARN


Copy link to this message
-
Re: question about when do resource matching in YARN
Harsh J 2013-09-24, 11:12
Yes, but the heartbeat coupling isn't necessary I think. One could
even use ZK write/watch approach for faster assignment of regular
work?

On Tue, Sep 24, 2013 at 2:24 PM, Steve Loughran <[EMAIL PROTECTED]> wrote:
> On 21 September 2013 09:19, Sandy Ryza <[EMAIL PROTECTED]> wrote:
>
>> I don't believe there is any reason scheduling decisions need to be coupled
>> with NodeManager heartbeats.  It doesn't sidestep any race conditions
>> because a NodeManager could die immediately after heartbeating.
>>
>>
> historically its been done for scale: you don't need the JT reaching out to
> 4K TT's just to give them work to do, instead let them connect in anyway
> and get work that way. And once they start reporting in completion then
> they can get given more work. It's very biased towards "worker nodes talk
> to the master" over "master approaches workers"
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

--
Harsh J