Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - out of memory for Reducer possible?


Copy link to this message
-
out of memory for Reducer possible?
Yang 2012-08-24, 01:43
if I do a

X = GROUP ALL my_relation;

in pig.

and let's say X is very big, and contains 1billion records,
in the reducer,would it be possible for the reducer to get OOM or similar
issues such as excessive spilling to disk?
---- assuming we do minimal processing in the reducer, and just loop over
each of the input record, doing no-op.
Thanks
Yang