Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: load properties from property file into Configuration object


Copy link to this message
-
Re: load properties from property file into Configuration object
Yes, you could use an XML file for that purpose (i.e. lots of config
options) and pass it along with -conf.

On Sat, Aug 3, 2013 at 6:32 PM, Juan Pino <[EMAIL PROTECTED]> wrote:
> Ok sorry, I think I got it: I would need to run my program like this:
>
> hadoop jar MYJAR MYCLASS -D property1=PROPERTY1 -D property2=PROPERTY2 -conf
> MYCONFIGFILE"
>
> Juan
>
>
> On Sat, Aug 3, 2013 at 1:46 PM, Juan Pino <[EMAIL PROTECTED]>
> wrote:
>>
>> Thanks Harsh for your reply.
>> Yes I think I am using the Tool framework ("public class HadoopJob extends
>> Configured implements Tool" + main function is
>> "System.exit(ToolRunner.run(new HadoopJob(), args));")
>>
>> "then you can pass -Dkey=val props directly into a config object at
>> runtime"
>> I'm a bit confused by this: I can pass -D key1=val1 -D key2=val2 etc. on
>> the command line, and with the Tool framework, those properties get
>> automatically loaded in the Configuration object, but if I have many of
>> these properties, how can I write them into a config file so that these
>> properties are loaded into the Configuration object ?
>>
>> Thanks very much,
>>
>> Juan
>>
>>
>> On Sat, Aug 3, 2013 at 1:20 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>
>>> This is a bit unnecessary to manually do. If you use the Tool
>>> framework, then you can pass -Dkey=val props directly into a config
>>> object at runtime. See
>>> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/util/Tool.html
>>> and
>>> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/util/GenericOptionsParser.html
>>>
>>> On Sat, Aug 3, 2013 at 5:30 PM, Juan Pino <[EMAIL PROTECTED]>
>>> wrote:
>>> > Hi,
>>> >
>>> > I am trying to simplify code that loads properties from a property file
>>> > into
>>> > a Configuration object:
>>> >
>>> > Short excerpt of the code:
>>> >
>>> > String configFile = args[0];
>>> > Properties p = new Properties();
>>> > p.load(new FileInputStream(configFile));
>>> > Configuration conf = getConf();
>>> > for (String prop: p.stringPropertyNames()) {
>>> >   conf.set(prop, p.getProperty(prop));
>>> > }
>>> >
>>> > The context is that my program can be run with an optional argument
>>> > which is
>>> > a config file as "hadoop jar MYJAR MYCLASS -D property1=PROPERTY1 -D
>>> > property2=PROPERTY2 [MYCONFIGFILE]"
>>> > The format of MYCONFIGFILE is
>>> > "property3=PROPERTY3<NEWLINE>property4=PROPERTY4 etc."
>>> >
>>> > Looking at the Configuration api, I didn't see any method that directly
>>> > loads properties from a property file.
>>> > Looking at the Configuration code, Configuration looks like a wrapper
>>> > around
>>> > Properties, so I thought there was a similar method to Properties.load
>>> > or a
>>> > method that sets several properties at the same time.
>>> > Is there a way to do this ?
>>> >
>>> > Thanks very much,
>>> >
>>> > Juan
>>> >
>>> > --
>>> > In light of the recent NSA scandal, please consider encrypting your
>>> > reply by
>>> > using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txt
>>> > If
>>> > you use webmail, you may consider mailvelope (http://mailvelope.com/)
>>> > which
>>> > is a very easy to use plugin available for chrome and probably soon for
>>> > firefox.
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>>
>> --
>> In light of the recent NSA scandal, please consider encrypting your reply
>> by using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txt
>> If you use webmail, you may consider mailvelope (http://mailvelope.com/)
>> which is a very easy to use plugin available for chrome and probably soon
>> for firefox.
>
>
>
>
> --
> In light of the recent NSA scandal, please consider encrypting your reply by
> using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txt If
> you use webmail, you may consider mailvelope (http://mailvelope.com/) which
> is a very easy to use plugin available for chrome and probably soon for
> firefox.

--
Harsh J