Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Udf to operate on a huge bag?


Copy link to this message
-
Re: Udf to operate on a huge bag?
Have you taken a look at the Algebraic and Accumulator interfaces? They
provide exactly these sorts of benefits.

2012/8/23 Yang <[EMAIL PROTECTED]>

> if I group records into a huge bag, and hand over to a Udf, would the input
> tuple actually
> create a bag with all the records? that way it may generate a OOM ??
>
> if indeed there is such an issue, I could probably implement the logic in
> plain pig, instead of Udf.
> but many times, logic is so complex that only Udf could do it.
>
>
> Thanks
> Yang
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB