Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - Queries on MRv2

Copy link to this message
Queries on MRv2
Praveen Sripati 2011-06-14, 13:01

I have gone through MapReduce NextGen Blog entries and JIRA and have the
following queries

>> There is a single API between the Scheduler and the ApplicationMaster:

>> (List <Container> newContainers, List <ContainerStatus>
containerStatuses) allocate (List <ResourceRequest> ask, List<Container>

>> The AM ask for specific resources via a list of ResourceRequests (ask)
and releases unnecessary Containers which were allocated by the Scheduler.

>> The response contains a list of newly allocated Containers and the
statuses of application-specific Containers that completed since the
previous interaction between the AM and the RM.

Q) If split-0 is is available in host1, host2 and host3, can
ApplicationMaster request a scheduler for a container on host1 or host2 or
host3? This way the scheduler can allocate the resources more effectively.

Q) In a cluster there might be nodes of different capacities, how will the
scheduler know that a particular node has 4 GB and another has 16 GB RAM
before allocating the resources to the ApplicationMaster?

Q) Are the unnecessary containers (List<Container> release) in the request
released by the ApplicationMaster the ones rejected by the ApplicationMaster
or those on which the map/reduce tasks have been completed?

Q) What does the following in the response contain - "List <ContainerStatus>

Q) Once the ApplicationMaster gets the list of the new containers from the
Scheduler, what is the interaction between the ApplicationMaster and the
Node Manager? Will the ApplicationMaster ask the Node Manager on the
different nodes to launch/monitor the map/reduce tasks in those containers?

Q) Does the Scheduler ask the Node Manager to create the containers on the
different nodes?

>> The resource requests are also aggregated by racks and then by the
special any (*) for all containers. All resource requests are subject to
change via the delta protocol.

Q) Does (*) mean that the ApplicationMaster is OK with a container in any
rack/host? This might be applicable for Reduce tasks.