Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Max tuples that can be handled by a reducer


Copy link to this message
-
Max tuples that can be handled by a reducer
Prasanth J 2012-07-26, 06:35
Hi

I wanted to find the maximum number of tuples a reducer can handle. For that I am using the following inside an UDF in a sampling job

maxTuples = Runtime.getRuntime.maxMemory() / tuple.getInMemorySize();

I am little skeptical about the maxMemory() usage as it will be different in sampling job and the actual job.
Does this always provide a good estimate of the max tuples a reducer can handle?

Thanks
-- Prasanth