Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> out of memory for Reducer possible?


Copy link to this message
-
out of memory for Reducer possible?
if I do a

X = GROUP ALL my_relation;

in pig.

and let's say X is very big, and contains 1billion records,
in the reducer,would it be possible for the reducer to get OOM or similar
issues such as excessive spilling to disk?
---- assuming we do minimal processing in the reducer, and just loop over
each of the input record, doing no-op.
Thanks
Yang
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB