Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Feed hdfs with external data.


Copy link to this message
-
Re: Feed hdfs with external data.
Hemanth Yamijala 2010-06-23, 09:19
Pierre,

> Ok, thank a lot about this, it worked out by specifying manually the conf...
> as in:
>
>            Configuration config = new Configuration();
>            config.set("dfs.name.dir", "/hd/name/");
>            config.set("dfs.data.dir", "/hd/data/");
>            config.set("fs.default.name", "hdfs://hadoop1/");
>
> Now, I would like to specify the conf dir location and have all this loaded
> automatically.
> Anyone?

You can use --config <path to conf dir> to your bin/hadoop commands. I
think it would also work if you set the HADOOP_CONF_DIR environment
variable to point to this path.

>
>
> On Wed, Jun 23, 2010 at 10:52 AM, Hemanth Yamijala <[EMAIL PROTECTED]>wrote:
>
>> Pierre,
>>
>> > I have a program that generates the data that's supposed to be treated by
>> > hadoop.
>> > It's a java program that should write right on hdfs.
>> > So as a test, I do this:
>> >
>> >
>> >
>> >            Configuration config = new Configuration();
>> >            FileSystem dfs = FileSystem.get(config);
>> >            dfs.mkdirs(new Path("/test"));
>> >
>> > Which of course failed, certainly because it's unaware of the position of
>> > the configuration files.
>> > How should I do this?
>> > Thanks.
>> >
>> >
>> > P.S. Just in case, here's my error:
>> >
>> > 10/06/23 10:04:06 ERROR conf.Configuration: Failed to set
>> > setXIncludeAware(true) for parser
>> > gnu.xml.dom.JAXPFactory@1d6096:java.lang.UnsupportedOperationException:
>> > This parser does not support specification "null" version "null"
>> > java.lang.UnsupportedOperationException: This parser does not support
>> > specification "null" version "null"
>> >        at
>> >
>> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
>> >        at
>>
>> This exception seems very similar to the one we faced and fixed using
>> the information here: http://bit.ly/a5zX1e. Could you give it a try ?
>>
>> Thanks
>> Hemanth
>>
>
>
>
> --
> http://www.neko-consulting.com
> Ego sum quis ego servo
> "Je suis ce que je protège"
> "I am what I protect"
>