Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Udf to operate on a huge bag?


Copy link to this message
-
Udf to operate on a huge bag?
Yang 2012-08-24, 02:00
if I group records into a huge bag, and hand over to a Udf, would the input
tuple actually
create a bag with all the records? that way it may generate a OOM ??

if indeed there is such an issue, I could probably implement the logic in
plain pig, instead of Udf.
but many times, logic is so complex that only Udf could do it.
Thanks
Yang