Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> Yarn NodeManager services


Copy link to this message
-
Re: Yarn NodeManager services
On Tue, Oct 11, 2011 at 2:05 PM,  <[EMAIL PROTECTED]> wrote:
> As part of MPI implementation in Yarn (aka Hamster), I was looking at
> refactoring some of the functionality into node manager services, so that
> it can be reused by other frameworks too. (Based on the discussion with
> some folks led me to believe that suffle etc is also being planned as NM
> service, separating it from deep integration with node manager.)

A general-purpose transfer protocol for intermediate data was designed
and scrapped for time. So was a local service allocated and shared by
resident containers. Instead, the NM supports loading modules (like
ShuffleHandler) at startup that implement auxiliary (or its
misspelling; my bad), ops-approved services, particularly shared
protocols independent of container and application allocation.

> I could not find the documentation for building NM services, in the MR-279
> architecture PDF. Does the service need to reside inside the NM jvm ?

Yes.

>  How
> does NM talk to the service ?

The NM loads and starts a list of services from the config by
classname. Each of these services follows the same lifecycle as the
other NM components. Each service is given a name in the config, which
is also its address for containers launched on that machine. Each
container can pass an opaque blob to the service to initialize it with
container-specific data. For the MR shuffle, these are the job
credentials.

Don't bother listing everything that's wrong with this model. Its
principal virtue was speed of implementation.

> How do containers discover the service ?

They don't. They need to know the name of the service a priori. -C

> (I
> know I can look at the code, but if there is already some docs (or even
> philosophical musings :-), it will make my life easier.
>
> Thanks,
>
> - milind
>
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)
>
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB