-Re: Yarn NodeManager services
Chris Douglas 2011-10-11, 21:33
On Tue, Oct 11, 2011 at 2:05 PM, <[EMAIL PROTECTED]> wrote:
> As part of MPI implementation in Yarn (aka Hamster), I was looking at
> refactoring some of the functionality into node manager services, so that
> it can be reused by other frameworks too. (Based on the discussion with
> some folks led me to believe that suffle etc is also being planned as NM
> service, separating it from deep integration with node manager.)
A general-purpose transfer protocol for intermediate data was designed
and scrapped for time. So was a local service allocated and shared by
resident containers. Instead, the NM supports loading modules (like
ShuffleHandler) at startup that implement auxiliary (or its
misspelling; my bad), ops-approved services, particularly shared
protocols independent of container and application allocation.
> I could not find the documentation for building NM services, in the MR-279
> architecture PDF. Does the service need to reside inside the NM jvm ?
> does NM talk to the service ?
The NM loads and starts a list of services from the config by
classname. Each of these services follows the same lifecycle as the
other NM components. Each service is given a name in the config, which
is also its address for containers launched on that machine. Each
container can pass an opaque blob to the service to initialize it with
container-specific data. For the MR shuffle, these are the job
Don't bother listing everything that's wrong with this model. Its
principal virtue was speed of implementation.
> How do containers discover the service ?
They don't. They need to know the name of the service a priori. -C
> know I can look at the code, but if there is already some docs (or even
> philosophical musings :-), it will make my life easier.
> - milind
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)