Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> out of memory for Reducer possible?


Copy link to this message
-
out of memory for Reducer possible?
if I do a

X = GROUP ALL my_relation;

in pig.

and let's say X is very big, and contains 1billion records,
in the reducer,would it be possible for the reducer to get OOM or similar
issues such as excessive spilling to disk?
---- assuming we do minimal processing in the reducer, and just loop over
each of the input record, doing no-op.
Thanks
Yang