Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Passing a BAG to Pig UDF constructor?


Copy link to this message
-
Passing a BAG to Pig UDF constructor?
Dexin Wang 2012-06-26, 17:27
Is it possible to pass a bag to a Pig UDF constructor?

Basically in the constructor I want to initialize some hash map so that on
every exec operation, I can use the hashmap to do a lookup and find the
value I need, and apply some algorithm to it.

I realize I could just do a replicated join to achieve similar things but
the algorithm is more than a few lines and there are some edge cases so I
would rather wrap that logic inside a UDF function. I also realize I could
just pass a file path to the constructor and read the files to initialize
the hashmap but my files are on Amazon's S3 and I don't want to deal with
S3 API to read the file.

Is this possible or is there some alternative ways to achieve the same
thing?

Thanks.
Dexin
+
Mridul Muralidharan 2012-06-26, 21:29
+
Jonathan Coveney 2012-06-26, 21:41
+
Mridul Muralidharan 2012-06-26, 21:50
+
Dexin Wang 2012-06-27, 17:30
+
Dexin Wang 2012-06-27, 23:08
+
Abhinav Neelam 2012-06-28, 09:54
+
Dexin Wang 2012-06-28, 18:46
+
Abhinav Neelam 2012-06-29, 05:17
+
Mridul Muralidharan 2012-06-29, 08:07
+
Jonathan Coveney 2012-06-29, 17:25