Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Udf to operate on a huge bag?


Copy link to this message
-
Udf to operate on a huge bag?
if I group records into a huge bag, and hand over to a Udf, would the input
tuple actually
create a bag with all the records? that way it may generate a OOM ??

if indeed there is such an issue, I could probably implement the logic in
plain pig, instead of Udf.
but many times, logic is so complex that only Udf could do it.
Thanks
Yang
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB