Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Question about properties for Loader


Copy link to this message
-
Re: Question about properties for Loader
Jeff Yuan 2013-02-25, 01:07
Thanks for the pointers Prashant. I will take a look at PigStorage.

I have a system for storing metadata, so users don't have to specify it.

With respect to the properties, I guess my question is, are the ones
passed in from the command line via -p stored in Property or
Configuration from the UDFContext? What's the difference between
Property and Configuration?

Thanks.

On Sun, Feb 24, 2013 at 4:02 PM, Prashant Kommireddi
<[EMAIL PROTECTED]> wrote:
> Hi Jeff,
>
> How do you see your loader being used? Would users specify schema file or
> would that be something your loader sets without user being aware of it?
> Can you pass it in as a constructor argument instead?
>
> UDFContext could be used, like you said to set/retrieve properties. You
> might want to take a look at PigStorage that does something very similar
> (look for the method applySchema(Tuple tup) )
>
> On Sun, Feb 24, 2013 at 3:33 PM, Jeff Yuan <[EMAIL PROTECTED]> wrote:
>
>> I'm trying to write a loader, extending LoadFunc, to read a specific
>> file format.
>>
>> My question, how do I pass properties to it (for example the schema of
>> the file type I'm loading)?  Would it be using the -p parameter from
>> the cmdline when issuing the query?
>>
>> The second part of the question is, how would I access the passed in
>> property/configuration from the code?  So far I'm theorizing it's
>> something like this:
>>         Properties p = udfc.getUDFProperties(this.getClass(), new
>> String[]{ contextSignature });
>>         Configuration conf = udfc.getJobConf();
>> Then get it from p or conf?
>>
>> Thanks a lot for any pointers.
>>
>> -Jeff
>>