Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Passing a BAG to Pig UDF constructor?


Copy link to this message
-
Passing a BAG to Pig UDF constructor?
Is it possible to pass a bag to a Pig UDF constructor?

Basically in the constructor I want to initialize some hash map so that on
every exec operation, I can use the hashmap to do a lookup and find the
value I need, and apply some algorithm to it.

I realize I could just do a replicated join to achieve similar things but
the algorithm is more than a few lines and there are some edge cases so I
would rather wrap that logic inside a UDF function. I also realize I could
just pass a file path to the constructor and read the files to initialize
the hashmap but my files are on Amazon's S3 and I don't want to deal with
S3 API to read the file.

Is this possible or is there some alternative ways to achieve the same
thing?

Thanks.
Dexin
+
Mridul Muralidharan 2012-06-26, 21:29
+
Jonathan Coveney 2012-06-26, 21:41
+
Mridul Muralidharan 2012-06-26, 21:50
+
Dexin Wang 2012-06-27, 17:30
+
Dexin Wang 2012-06-27, 23:08
+
Abhinav Neelam 2012-06-28, 09:54
+
Dexin Wang 2012-06-28, 18:46
+
Abhinav Neelam 2012-06-29, 05:17
+
Mridul Muralidharan 2012-06-29, 08:07
+
Jonathan Coveney 2012-06-29, 17:25
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB