Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: core-site.xml file is being ignored by new Configuration()


Copy link to this message
-
Re: core-site.xml file is being ignored by new Configuration()
I am not sure you actually test if the file exists in your code:
System.out.println("File exists : " + theXMLFile.length());
This is really only the file path.

You could try to load the file to make sure:

File file = new File(theXMLFile);

// Test if exists

// ...

// Load in ressource, I had issues loading confs and found this way is
working in all my cases, including custom FS in fs.default.name

conf.addResource(file.getAbsoluteFile().toURI().toURL());
(This is not really optimized but I believe it is ok for a JUnit setup
phase)

Julien

2013/1/17 Jay Vyas <[EMAIL PROTECTED]>

> no difference.
>
> This is really odd.
>
> The only way I can get it to work is by the following:
>
>
> /**
>  * A simple test harness for glusterfs
>  * @author jvyas
>  */
> public class TestMapReducerGlusterFS {
>
>
>     @Test
>     public void test() throws Exception{
>         Configuration conf = new Configuration(false);
>         System.out.println("before reload:"+conf);
>         //necessary?
>         conf.set("fs.default.name", ".......");
>         conf.set("fs.blah.mount", "/mnt/......");
>         conf.set("fs.blahserver", "192.168.1.36");
>         conf.set("iomod","Off");
>          ...
>     }
> }
>
> Thinking that this is a little known issue in the current hadoop
> parser/configurator? because very rarely do people write unit tests which
> utilize abnormal ( i.e. non-local or non-hdfs) file systems.
>
>
> On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <[EMAIL PROTECTED]> wrote:
>
>> Result any different if you call reloadConfiguration()?
>>
>> Thanks,
>>
>> Tom
>>
>> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
>> > Hi guys:
>> >
>> > I've finally extracted my problem of loading a special filesystem
>> > into a unit test.
>> >
>> > Below, clearly, Im creating a raw configuration and adding a single
>> resource
>> > to it (core-site.xml).
>> >
>> > Afterwards, i print to confirm that the file exists.
>> >
>> > Finally, I try to create a FileSystem from that file.  However, no file
>> > system is created :(.
>> >
>> > Any thoughts on why this core-site.xml file is being ignoreed?
>> > To note: I've also tried putting core-site.xml in the working
>> directory, and
>> > in a local conf/ directory.  This absolute path was a last effort to
>> really
>> > confirm that the local path wasn't somehow being mangled by the junit
>> > runtime.
>> >
>> > @Test
>> >     public void test() throws Exception{
>> >         Configuration conf = new Configuration(false);
>> >         System.out.println("b4:"+conf);
>> >         String
>> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
>> >         conf.addResource(theXMLFile);
>> >         System.out.println("File exists : " + theXMLFile.length());
>> >         System.out.println("after:"+conf);
>> >         System.out.println("file sytem " + conf.get("fs.default.name
>> "));
>> >         conf.writeXml(System.out);
>> >
>> >
>> >         System.out.println("Working directory = " +
>> > System.getProperty("user.dir"));
>> >         org.apache.hadoop.fs.FileSystem fs >> > org.apache.hadoop.fs.FileSystem.get(conf);
>> >         System.out.println("before test filesystem = " + fs);
>> >         //WordCount.main(new
>> > String[]{"resources/wordcount.txt","wordcountout.txt"});
>> >     }
>> >
>> > --
>> > Jay Vyas
>> > http://jayunit100.blogspot.com
>>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB