Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> SpillableMemoryManager - low memory handler called


Copy link to this message
-
Re: SpillableMemoryManager - low memory handler called
0.6

Sent from my iPhone

On May 6, 2010, at 12:16 PM, "Olga Natkovich" <[EMAIL PROTECTED]>  
wrote:

> Which version of Pig are you using?
>
> -----Original Message-----
> From: Corbin Hoenes [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, May 06, 2010 10:29 AM
> To: [EMAIL PROTECTED]
> Subject: SpillableMemoryManager - low memory handler called
>
> Hi Piggers - Seeing an issue with a particular script where our job is
> taking 6hrs 42min to complete.
>
> syslogs are showing loads of these:
> INFO : org.apache.pig.impl.util.SpillableMemoryManager - low memory
> handler called (Usage threshold exceeded) init = 5439488(5312K) used > 283443200(276800K) committed = 357957632(349568K) max > 357957632(349568K)
> INFO : org.apache.pig.impl.util.SpillableMemoryManager - low memory
> handler called (Usage threshold exceeded) init = 5439488(5312K) used > 267128840(260868K) committed = 357957632(349568K) max > 357957632(349568K)
> One iteresting thing is it's the map phase that is slow and one of the
> mappers is getting 8GB of input while the other 2000 or so mappers are
> getting MBs and hundreds of MBs of data.
>
> Any where I can start looking?
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB