Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - SpillableMemoryManager - low memory handler called


Copy link to this message
-
SpillableMemoryManager - low memory handler called
Corbin Hoenes 2010-05-06, 17:28
Hi Piggers - Seeing an issue with a particular script where our job is taking 6hrs 42min to complete.  

syslogs are showing loads of these:
INFO : org.apache.pig.impl.util.SpillableMemoryManager - low memory handler called (Usage threshold exceeded) init = 5439488(5312K) used = 283443200(276800K) committed = 357957632(349568K) max = 357957632(349568K)
INFO : org.apache.pig.impl.util.SpillableMemoryManager - low memory handler called (Usage threshold exceeded) init = 5439488(5312K) used = 267128840(260868K) committed = 357957632(349568K) max = 357957632(349568K)
One iteresting thing is it's the map phase that is slow and one of the mappers is getting 8GB of input while the other 2000 or so mappers are getting MBs and hundreds of MBs of data.

Any where I can start looking?