Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Schema access while writing


Copy link to this message
-
Schema access while writing
Hello,

I'm implementing a custom Store UDF using StoreFuncInterface.

I need access to the ResourceSchema object each time I do a putNext
operation, but am unable to do this since checkSchema() [which carries
what I require] is only called once and that's during job init or so.
If I try to store a reference/copy of that object, it does not work
since the mapper-side instances of my UDF don't get the checkSchema()
call.

What I've tried is to process and store required parts of the
ResourceSchema into the Job's Configuration using the fact that
setStoreLocation(String, Job) is called in the init AFTER
checkSchema(), and tried retrieving that to no avail. It looks like
the changes I make to the Job object given to me goes futile, as I get
a null at the map/reduce side for the configuration name I've stored
it as.

What do I do to access either the ResourceSchema or even a
Job-Configuration variable that I wish to set post processing the
ResourceSchema?

--
Harsh J
www.harshj.com
+
Scott Carey 2010-06-28, 14:54
+
Harsh J 2010-06-28, 16:08
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB