Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Question about properties for Loader

Jeff Yuan 2013-02-24, 23:33
Prashant Kommireddi 2013-02-25, 00:02
Copy link to this message
Re: Question about properties for Loader
Thanks for the pointers Prashant. I will take a look at PigStorage.

I have a system for storing metadata, so users don't have to specify it.

With respect to the properties, I guess my question is, are the ones
passed in from the command line via -p stored in Property or
Configuration from the UDFContext? What's the difference between
Property and Configuration?


On Sun, Feb 24, 2013 at 4:02 PM, Prashant Kommireddi
> Hi Jeff,
> How do you see your loader being used? Would users specify schema file or
> would that be something your loader sets without user being aware of it?
> Can you pass it in as a constructor argument instead?
> UDFContext could be used, like you said to set/retrieve properties. You
> might want to take a look at PigStorage that does something very similar
> (look for the method applySchema(Tuple tup) )
> On Sun, Feb 24, 2013 at 3:33 PM, Jeff Yuan <[EMAIL PROTECTED]> wrote:
>> I'm trying to write a loader, extending LoadFunc, to read a specific
>> file format.
>> My question, how do I pass properties to it (for example the schema of
>> the file type I'm loading)?  Would it be using the -p parameter from
>> the cmdline when issuing the query?
>> The second part of the question is, how would I access the passed in
>> property/configuration from the code?  So far I'm theorizing it's
>> something like this:
>>         Properties p = udfc.getUDFProperties(this.getClass(), new
>> String[]{ contextSignature });
>>         Configuration conf = udfc.getJobConf();
>> Then get it from p or conf?
>> Thanks a lot for any pointers.
>> -Jeff
Dmitriy Ryaboy 2013-02-25, 03:24