Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Max tuples that can be handled by a reducer


Copy link to this message
-
Max tuples that can be handled by a reducer
Hi

I wanted to find the maximum number of tuples a reducer can handle. For that I am using the following inside an UDF in a sampling job

maxTuples = Runtime.getRuntime.maxMemory() / tuple.getInMemorySize();

I am little skeptical about the maxMemory() usage as it will be different in sampling job and the actual job.
Does this always provide a good estimate of the max tuples a reducer can handle?

Thanks
-- Prasanth

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB