Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Feed hdfs with external data.


Copy link to this message
-
Re: Feed hdfs with external data.
Sorry, I might have misunderstood... So, you have a Java application
that needs to use this configuration. In that case, I think setting
the directory containing the config files into the classpath of your
application should work fine.

Thanks
Hemanth

On Wed, Jun 23, 2010 at 2:49 PM, Hemanth Yamijala <[EMAIL PROTECTED]> wrote:
> Pierre,
>
>> Ok, thank a lot about this, it worked out by specifying manually the conf...
>> as in:
>>
>>            Configuration config = new Configuration();
>>            config.set("dfs.name.dir", "/hd/name/");
>>            config.set("dfs.data.dir", "/hd/data/");
>>            config.set("fs.default.name", "hdfs://hadoop1/");
>>
>> Now, I would like to specify the conf dir location and have all this loaded
>> automatically.
>> Anyone?
>
> You can use --config <path to conf dir> to your bin/hadoop commands. I
> think it would also work if you set the HADOOP_CONF_DIR environment
> variable to point to this path.
>
>>
>>
>> On Wed, Jun 23, 2010 at 10:52 AM, Hemanth Yamijala <[EMAIL PROTECTED]>wrote:
>>
>>> Pierre,
>>>
>>> > I have a program that generates the data that's supposed to be treated by
>>> > hadoop.
>>> > It's a java program that should write right on hdfs.
>>> > So as a test, I do this:
>>> >
>>> >
>>> >
>>> >            Configuration config = new Configuration();
>>> >            FileSystem dfs = FileSystem.get(config);
>>> >            dfs.mkdirs(new Path("/test"));
>>> >
>>> > Which of course failed, certainly because it's unaware of the position of
>>> > the configuration files.
>>> > How should I do this?
>>> > Thanks.
>>> >
>>> >
>>> > P.S. Just in case, here's my error:
>>> >
>>> > 10/06/23 10:04:06 ERROR conf.Configuration: Failed to set
>>> > setXIncludeAware(true) for parser
>>> > gnu.xml.dom.JAXPFactory@1d6096:java.lang.UnsupportedOperationException:
>>> > This parser does not support specification "null" version "null"
>>> > java.lang.UnsupportedOperationException: This parser does not support
>>> > specification "null" version "null"
>>> >        at
>>> >
>>> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
>>> >        at
>>>
>>> This exception seems very similar to the one we faced and fixed using
>>> the information here: http://bit.ly/a5zX1e. Could you give it a try ?
>>>
>>> Thanks
>>> Hemanth
>>>
>>
>>
>>
>> --
>> http://www.neko-consulting.com
>> Ego sum quis ego servo
>> "Je suis ce que je protège"
>> "I am what I protect"
>>
>