Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Feed hdfs with external data.


Copy link to this message
-
Re: Feed hdfs with external data.
Pierre,

> Ok, thank a lot about this, it worked out by specifying manually the conf...
> as in:
>
>            Configuration config = new Configuration();
>            config.set("dfs.name.dir", "/hd/name/");
>            config.set("dfs.data.dir", "/hd/data/");
>            config.set("fs.default.name", "hdfs://hadoop1/");
>
> Now, I would like to specify the conf dir location and have all this loaded
> automatically.
> Anyone?

You can use --config <path to conf dir> to your bin/hadoop commands. I
think it would also work if you set the HADOOP_CONF_DIR environment
variable to point to this path.

>
>
> On Wed, Jun 23, 2010 at 10:52 AM, Hemanth Yamijala <[EMAIL PROTECTED]>wrote:
>
>> Pierre,
>>
>> > I have a program that generates the data that's supposed to be treated by
>> > hadoop.
>> > It's a java program that should write right on hdfs.
>> > So as a test, I do this:
>> >
>> >
>> >
>> >            Configuration config = new Configuration();
>> >            FileSystem dfs = FileSystem.get(config);
>> >            dfs.mkdirs(new Path("/test"));
>> >
>> > Which of course failed, certainly because it's unaware of the position of
>> > the configuration files.
>> > How should I do this?
>> > Thanks.
>> >
>> >
>> > P.S. Just in case, here's my error:
>> >
>> > 10/06/23 10:04:06 ERROR conf.Configuration: Failed to set
>> > setXIncludeAware(true) for parser
>> > gnu.xml.dom.JAXPFactory@1d6096:java.lang.UnsupportedOperationException:
>> > This parser does not support specification "null" version "null"
>> > java.lang.UnsupportedOperationException: This parser does not support
>> > specification "null" version "null"
>> >        at
>> >
>> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
>> >        at
>>
>> This exception seems very similar to the one we faced and fixed using
>> the information here: http://bit.ly/a5zX1e. Could you give it a try ?
>>
>> Thanks
>> Hemanth
>>
>
>
>
> --
> http://www.neko-consulting.com
> Ego sum quis ego servo
> "Je suis ce que je protège"
> "I am what I protect"
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB