Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Question about properties for Loader


+
Jeff Yuan 2013-02-24, 23:33
+
Prashant Kommireddi 2013-02-25, 00:02
Copy link to this message
-
Re: Question about properties for Loader
Thanks for the pointers Prashant. I will take a look at PigStorage.

I have a system for storing metadata, so users don't have to specify it.

With respect to the properties, I guess my question is, are the ones
passed in from the command line via -p stored in Property or
Configuration from the UDFContext? What's the difference between
Property and Configuration?

Thanks.

On Sun, Feb 24, 2013 at 4:02 PM, Prashant Kommireddi
<[EMAIL PROTECTED]> wrote:
> Hi Jeff,
>
> How do you see your loader being used? Would users specify schema file or
> would that be something your loader sets without user being aware of it?
> Can you pass it in as a constructor argument instead?
>
> UDFContext could be used, like you said to set/retrieve properties. You
> might want to take a look at PigStorage that does something very similar
> (look for the method applySchema(Tuple tup) )
>
> On Sun, Feb 24, 2013 at 3:33 PM, Jeff Yuan <[EMAIL PROTECTED]> wrote:
>
>> I'm trying to write a loader, extending LoadFunc, to read a specific
>> file format.
>>
>> My question, how do I pass properties to it (for example the schema of
>> the file type I'm loading)?  Would it be using the -p parameter from
>> the cmdline when issuing the query?
>>
>> The second part of the question is, how would I access the passed in
>> property/configuration from the code?  So far I'm theorizing it's
>> something like this:
>>         Properties p = udfc.getUDFProperties(this.getClass(), new
>> String[]{ contextSignature });
>>         Configuration conf = udfc.getJobConf();
>> Then get it from p or conf?
>>
>> Thanks a lot for any pointers.
>>
>> -Jeff
>>
+
Dmitriy Ryaboy 2013-02-25, 03:24
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB