Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: load properties from property file into Configuration object


Copy link to this message
-
Re: load properties from property file into Configuration object
Yes, you could use an XML file for that purpose (i.e. lots of config
options) and pass it along with -conf.

On Sat, Aug 3, 2013 at 6:32 PM, Juan Pino <[EMAIL PROTECTED]> wrote:
> Ok sorry, I think I got it: I would need to run my program like this:
>
> hadoop jar MYJAR MYCLASS -D property1=PROPERTY1 -D property2=PROPERTY2 -conf
> MYCONFIGFILE"
>
> Juan
>
>
> On Sat, Aug 3, 2013 at 1:46 PM, Juan Pino <[EMAIL PROTECTED]>
> wrote:
>>
>> Thanks Harsh for your reply.
>> Yes I think I am using the Tool framework ("public class HadoopJob extends
>> Configured implements Tool" + main function is
>> "System.exit(ToolRunner.run(new HadoopJob(), args));")
>>
>> "then you can pass -Dkey=val props directly into a config object at
>> runtime"
>> I'm a bit confused by this: I can pass -D key1=val1 -D key2=val2 etc. on
>> the command line, and with the Tool framework, those properties get
>> automatically loaded in the Configuration object, but if I have many of
>> these properties, how can I write them into a config file so that these
>> properties are loaded into the Configuration object ?
>>
>> Thanks very much,
>>
>> Juan
>>
>>
>> On Sat, Aug 3, 2013 at 1:20 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>
>>> This is a bit unnecessary to manually do. If you use the Tool
>>> framework, then you can pass -Dkey=val props directly into a config
>>> object at runtime. See
>>> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/util/Tool.html
>>> and
>>> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/util/GenericOptionsParser.html
>>>
>>> On Sat, Aug 3, 2013 at 5:30 PM, Juan Pino <[EMAIL PROTECTED]>
>>> wrote:
>>> > Hi,
>>> >
>>> > I am trying to simplify code that loads properties from a property file
>>> > into
>>> > a Configuration object:
>>> >
>>> > Short excerpt of the code:
>>> >
>>> > String configFile = args[0];
>>> > Properties p = new Properties();
>>> > p.load(new FileInputStream(configFile));
>>> > Configuration conf = getConf();
>>> > for (String prop: p.stringPropertyNames()) {
>>> >   conf.set(prop, p.getProperty(prop));
>>> > }
>>> >
>>> > The context is that my program can be run with an optional argument
>>> > which is
>>> > a config file as "hadoop jar MYJAR MYCLASS -D property1=PROPERTY1 -D
>>> > property2=PROPERTY2 [MYCONFIGFILE]"
>>> > The format of MYCONFIGFILE is
>>> > "property3=PROPERTY3<NEWLINE>property4=PROPERTY4 etc."
>>> >
>>> > Looking at the Configuration api, I didn't see any method that directly
>>> > loads properties from a property file.
>>> > Looking at the Configuration code, Configuration looks like a wrapper
>>> > around
>>> > Properties, so I thought there was a similar method to Properties.load
>>> > or a
>>> > method that sets several properties at the same time.
>>> > Is there a way to do this ?
>>> >
>>> > Thanks very much,
>>> >
>>> > Juan
>>> >
>>> > --
>>> > In light of the recent NSA scandal, please consider encrypting your
>>> > reply by
>>> > using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txt
>>> > If
>>> > you use webmail, you may consider mailvelope (http://mailvelope.com/)
>>> > which
>>> > is a very easy to use plugin available for chrome and probably soon for
>>> > firefox.
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>>
>> --
>> In light of the recent NSA scandal, please consider encrypting your reply
>> by using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txt
>> If you use webmail, you may consider mailvelope (http://mailvelope.com/)
>> which is a very easy to use plugin available for chrome and probably soon
>> for firefox.
>
>
>
>
> --
> In light of the recent NSA scandal, please consider encrypting your reply by
> using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txt If
> you use webmail, you may consider mailvelope (http://mailvelope.com/) which
> is a very easy to use plugin available for chrome and probably soon for
> firefox.

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB