Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Passing a BAG to Pig UDF constructor?

Copy link to this message
Passing a BAG to Pig UDF constructor?
Is it possible to pass a bag to a Pig UDF constructor?

Basically in the constructor I want to initialize some hash map so that on
every exec operation, I can use the hashmap to do a lookup and find the
value I need, and apply some algorithm to it.

I realize I could just do a replicated join to achieve similar things but
the algorithm is more than a few lines and there are some edge cases so I
would rather wrap that logic inside a UDF function. I also realize I could
just pass a file path to the constructor and read the files to initialize
the hashmap but my files are on Amazon's S3 and I don't want to deal with
S3 API to read the file.

Is this possible or is there some alternative ways to achieve the same

Mridul Muralidharan 2012-06-26, 21:29
Jonathan Coveney 2012-06-26, 21:41
Mridul Muralidharan 2012-06-26, 21:50
Dexin Wang 2012-06-27, 17:30
Dexin Wang 2012-06-27, 23:08
Abhinav Neelam 2012-06-28, 09:54
Dexin Wang 2012-06-28, 18:46
Abhinav Neelam 2012-06-29, 05:17
Mridul Muralidharan 2012-06-29, 08:07
Jonathan Coveney 2012-06-29, 17:25