Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> Queries on MRv2


Copy link to this message
-
Re: Queries on MRv2
Arun,

Thanks for the reply.

Q) What happens if an ApplicationMaster asks a NM to launch a container and
then releases the container in the allocate call later?

Q) So, the NM watches the UNIX Process/Containers and sends the status to
the ApplicationManager. Later the ApplicationManager sends the status of the
containers in response to the allocate call to the ApplicationMaster. Why
should the ApplicationMaster be aware of the container status, since it's
already tracking the map/reduce tasks in the containers?

Q) Does the ApplicationMaster notify the NodeManager to exit the UNIX
Process when the map/reduce tasks in that particular container are
completed? Are the containers re-used?

Q) The ApplicationManager asks the NodeManager to create a container and
also launch the map/reduce task in it. From then on the ApplicationManager
and Map/Reduce tasks interact directly without the NodeManager. Am I
correct?

Praveen

On Wed, Jun 15, 2011 at 12:59 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

>
> On Jun 14, 2011, at 6:31 PM, Praveen Sripati wrote:
>
>  Hi,
>>
>> I have gone through MapReduce NextGen Blog entries and JIRA and have the
>> following queries
>>
>>  There is a single API between the Scheduler and the ApplicationMaster:
>>>>
>>>
>>  (List <Container> newContainers, List <ContainerStatus>
>>>>
>>> containerStatuses) allocate (List <ResourceRequest> ask, List<Container>
>> release)
>>
>>  The AM ask for specific resources via a list of ResourceRequests (ask)
>>>>
>>> and releases unnecessary Containers which were allocated by the
>> Scheduler.
>>
>>  The response contains a list of newly allocated Containers and the
>>>>
>>> statuses of application-specific Containers that completed since the
>> previous interaction between the AM and the RM.
>>
>> Q) If split-0 is is available in host1, host2 and host3, can
>> ApplicationMaster request a scheduler for a container on host1 or host2 or
>> host3? This way the scheduler can allocate the resources more effectively.
>>
>>
> Yes, absolutely.
>
>
>  Q) In a cluster there might be nodes of different capacities, how will the
>> scheduler know that a particular node has 4 GB and another has 16 GB RAM
>> before allocating the resources to the ApplicationMaster?
>>
>>
> The NodeManager informs the RM about its capabilities on registration. The
> RM allocates appropriate resources to the AM(s).
>
>
>  Q) Are the unnecessary containers (List<Container> release) in the request
>> released by the ApplicationMaster the ones rejected by the
>> ApplicationMaster
>> or those on which the map/reduce tasks have been completed?
>>
>>
> Only unused ones.
>
>
>  Q) What does the following in the response contain - "List
>> <ContainerStatus>
>> containerStatuses"?
>>
>>
> Status for completed completed containers.
>
>
>  Q) Once the ApplicationMaster gets the list of the new containers from the
>> Scheduler, what is the interaction between the ApplicationMaster and the
>> Node Manager? Will the ApplicationMaster ask the Node Manager on the
>> different nodes to launch/monitor the map/reduce tasks in those
>> containers?
>>
>>
> No, the AM directly monitors the containers via an application-specific
> protocol.
>
> For MR applications we use TaskUmbilicalProtocol.
>
> The NM just monitors the unix process and informs the RM on exit of the
> unix process.
>
>
>  Q) Does the Scheduler ask the Node Manager to create the containers on the
>> different nodes?
>>
>
> No, the Scheduler allocates them to the respective AMs who then launch the
> container by talking to the NM.
>
> The NM can securely verify the authenticity of the 'container launch'
> request, including the resources allocated to the container.
>
>
>
>>  The resource requests are also aggregated by racks and then by the
>>>>
>>> special any (*) for all containers. All resource requests are subject to
>> change via the delta protocol.
>>
>> Q) Does (*) mean that the ApplicationMaster is OK with a container in any
>> rack/host? This might be applicable for Reduce tasks.