|
|
-
out of memory for Reducer possible?Yang 2012-08-24, 01:43
if I do a
X = GROUP ALL my_relation; in pig. and let's say X is very big, and contains 1billion records, in the reducer,would it be possible for the reducer to get OOM or similar issues such as excessive spilling to disk? ---- assuming we do minimal processing in the reducer, and just loop over each of the input record, doing no-op. Thanks Yang |