Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - SpillableMemoryManager - low memory handler called


Copy link to this message
-
Re: SpillableMemoryManager - low memory handler called
Corbin Hoenes 2010-05-06, 18:31
0.6

Sent from my iPhone

On May 6, 2010, at 12:16 PM, "Olga Natkovich" <[EMAIL PROTECTED]>  
wrote:

> Which version of Pig are you using?
>
> -----Original Message-----
> From: Corbin Hoenes [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, May 06, 2010 10:29 AM
> To: [EMAIL PROTECTED]
> Subject: SpillableMemoryManager - low memory handler called
>
> Hi Piggers - Seeing an issue with a particular script where our job is
> taking 6hrs 42min to complete.
>
> syslogs are showing loads of these:
> INFO : org.apache.pig.impl.util.SpillableMemoryManager - low memory
> handler called (Usage threshold exceeded) init = 5439488(5312K) used > 283443200(276800K) committed = 357957632(349568K) max > 357957632(349568K)
> INFO : org.apache.pig.impl.util.SpillableMemoryManager - low memory
> handler called (Usage threshold exceeded) init = 5439488(5312K) used > 267128840(260868K) committed = 357957632(349568K) max > 357957632(349568K)
> One iteresting thing is it's the map phase that is slow and one of the
> mappers is getting 8GB of input while the other 2000 or so mappers are
> getting MBs and hundreds of MBs of data.
>
> Any where I can start looking?
>
>