Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Avoid Killing of YARN container on excess virtual memory usage


Copy link to this message
-
Re: Avoid Killing of YARN container on excess virtual memory usage
Krishna Kishore Bonagiri 2012-10-19, 01:47
Thanks Harsha, I was actually looking this part of the code in
ContainerMonitorImpl.java trying to find how can I make use of it, but as
you said it's a bug and doesn't seem to be really handling the case.

Thanks,
Kishore

On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> >   Is there a way we can ask the YARN RM for not killing a container when
> it
> > uses excess virtual memory than the maximum it can use as per the
> > specification in the configuration file yarn-site.xml? We can't always
> > estimate the amount of virtual memory needed for our application running
> on
> > a container, but we don't want to get it killed in a case it exceeds the
> > maximum limit.
> >
> >   Please suggest as to how can we come across this issue.
> >
> > Thanks,
> > Kishore
>
>
>
> --
> Harsh J
>