if I group records into a huge bag, and hand over to a Udf, would the input
tuple actually
create a bag with all the records? that way it may generate a OOM ??

if indeed there is such an issue, I could probably implement the logic in
plain pig, instead of Udf.
but many times, logic is so complex that only Udf could do it.
Thanks
Yang

Have you taken a look at the Algebraic and Accumulator interfaces? They
provide exactly these sorts of benefits.

2012/8/23 Yang <[EMAIL PROTECTED]>

> if I group records into a huge bag, and hand over to a Udf, would the input
> tuple actually
> create a bag with all the records? that way it may generate a OOM ??
>
> if indeed there is such an issue, I could probably implement the logic in
> plain pig, instead of Udf.
> but many times, logic is so complex that only Udf could do it.
>
>
> Thanks
> Yang
>