-out of memory for Reducer possible?
Yang 2012-08-24, 01:43
if I do a
X = GROUP ALL my_relation;
and let's say X is very big, and contains 1billion records,
in the reducer,would it be possible for the reducer to get OOM or similar
issues such as excessive spilling to disk?
---- assuming we do minimal processing in the reducer, and just loop over
each of the input record, doing no-op.