Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?


Copy link to this message
-
Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
Hi Jay,
Can you point out a detail about 1.x that job tracker having an
ever-expanding hash map. I want fix it.
On Tue, Dec 3, 2013 at 3:08 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> version is rewally important here..
>
> - If 1.x, then Where (NN , JT , TT ?)
> - if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since
> the resources are ephemeral.
>
> I know that some older 1x versions had an issue with the jobtracker having
> an ever-expanding hashmap or something like that, so that if you ran 100s
> of jobs, you could get OOM erros on the JobTracker.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB