Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - Queries on MRv2


Copy link to this message
-
Re: Queries on MRv2
Praveen Sripati 2011-06-15, 03:38
Mahadev,

MapReduce ApplicationMaster might behave well, but what about custom
ApplicationMasters for other models.

> Q) What happens if an ApplicationMaster asks a NM to launch a container
and
> then releases the container in the allocate call later?

> A) The Application Master only releases the container once the container
is done.

Thanks,
Praveen

On Wed, Jun 15, 2011 at 8:59 AM, Mahadev Konar <[EMAIL PROTECTED]> wrote:

> Praveen,
>  Answers in line:
>
> >
> > Q) What happens if an ApplicationMaster asks a NM to launch a container
> and
> > then releases the container in the allocate call later?
>
> The Application Master only releases the container once the container is
> done.
>
> >
> > Q) So, the NM watches the UNIX Process/Containers and sends the status to
> > the ApplicationManager. Later the ApplicationManager sends the status of
> the
> > containers in response to the allocate call to the ApplicationMaster. Why
> > should the ApplicationMaster be aware of the container status, since it's
> > already tracking the map/reduce tasks in the containers?
>
> Its just a way to notify the application master as soon as possible
> when the containers fail.
> This helps in speeding up the notification of failed containers else
> AM has to wait for discovering
> failures via timeouts.
>
> >
> > Q) Does the ApplicationMaster notify the NodeManager to exit the UNIX
> > Process when the map/reduce tasks in that particular container are
> > completed? Are the containers re-used?
>
> Yes it notifes the NM.
>
> Containers are not re used as of now. In future we do see the
> containers being re used but we'll need leases to do that.
>
> >
> > Q) The ApplicationManager asks the NodeManager to create a container and
> > also launch the map/reduce task in it. From then on the
> ApplicationManager
> > and Map/Reduce tasks interact directly without the NodeManager. Am I
> > correct?
> >
> I think you mean ApplicationMaster. Yes, the applicationmaster and
> map/reduce tasks talk directly
> without NM being involved.
>
> > Praveen
> >
> > On Wed, Jun 15, 2011 at 12:59 AM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >
> >>
> >> On Jun 14, 2011, at 6:31 PM, Praveen Sripati wrote:
> >>
> >>  Hi,
> >>>
> >>> I have gone through MapReduce NextGen Blog entries and JIRA and have
> the
> >>> following queries
> >>>
> >>>  There is a single API between the Scheduler and the ApplicationMaster:
> >>>>>
> >>>>
> >>>  (List <Container> newContainers, List <ContainerStatus>
> >>>>>
> >>>> containerStatuses) allocate (List <ResourceRequest> ask,
> List<Container>
> >>> release)
> >>>
> >>>  The AM ask for specific resources via a list of ResourceRequests (ask)
> >>>>>
> >>>> and releases unnecessary Containers which were allocated by the
> >>> Scheduler.
> >>>
> >>>  The response contains a list of newly allocated Containers and the
> >>>>>
> >>>> statuses of application-specific Containers that completed since the
> >>> previous interaction between the AM and the RM.
> >>>
> >>> Q) If split-0 is is available in host1, host2 and host3, can
> >>> ApplicationMaster request a scheduler for a container on host1 or host2
> or
> >>> host3? This way the scheduler can allocate the resources more
> effectively.
> >>>
> >>>
> >> Yes, absolutely.
> >>
> >>
> >>  Q) In a cluster there might be nodes of different capacities, how will
> the
> >>> scheduler know that a particular node has 4 GB and another has 16 GB
> RAM
> >>> before allocating the resources to the ApplicationMaster?
> >>>
> >>>
> >> The NodeManager informs the RM about its capabilities on registration.
> The
> >> RM allocates appropriate resources to the AM(s).
> >>
> >>
> >>  Q) Are the unnecessary containers (List<Container> release) in the
> request
> >>> released by the ApplicationMaster the ones rejected by the
> >>> ApplicationMaster
> >>> or those on which the map/reduce tasks have been completed?
> >>>
> >>>
> >> Only unused ones.
> >>
> >>
> >>  Q) What does the following in the response contain - "List