Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Avoid Killing of YARN container on excess virtual memory usage


+
Krishna Kishore Bonagiri 2012-10-18, 10:30
+
Harsh J 2012-10-18, 11:37
+
Harsh J 2012-10-18, 14:46
Copy link to this message
-
Re: Avoid Killing of YARN container on excess virtual memory usage
Thanks Harsha, I was actually looking this part of the code in
ContainerMonitorImpl.java trying to find how can I make use of it, but as
you said it's a bug and doesn't seem to be really handling the case.

Thanks,
Kishore

On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> >   Is there a way we can ask the YARN RM for not killing a container when
> it
> > uses excess virtual memory than the maximum it can use as per the
> > specification in the configuration file yarn-site.xml? We can't always
> > estimate the amount of virtual memory needed for our application running
> on
> > a container, but we don't want to get it killed in a case it exceeds the
> > maximum limit.
> >
> >   Please suggest as to how can we come across this issue.
> >
> > Thanks,
> > Kishore
>
>
>
> --
> Harsh J
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB