Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Schema access while writing

Copy link to this message
Schema access while writing

I'm implementing a custom Store UDF using StoreFuncInterface.

I need access to the ResourceSchema object each time I do a putNext
operation, but am unable to do this since checkSchema() [which carries
what I require] is only called once and that's during job init or so.
If I try to store a reference/copy of that object, it does not work
since the mapper-side instances of my UDF don't get the checkSchema()

What I've tried is to process and store required parts of the
ResourceSchema into the Job's Configuration using the fact that
setStoreLocation(String, Job) is called in the init AFTER
checkSchema(), and tried retrieving that to no avail. It looks like
the changes I make to the Job object given to me goes futile, as I get
a null at the map/reduce side for the configuration name I've stored
it as.

What do I do to access either the ResourceSchema or even a
Job-Configuration variable that I wish to set post processing the

Harsh J
Scott Carey 2010-06-28, 14:54
Harsh J 2010-06-28, 16:08