Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Re: core-site.xml file is being ignored by new Configuration()


Copy link to this message
-
Re: core-site.xml file is being ignored by new Configuration()
Julien Muller 2013-01-17, 10:06
I am not sure you actually test if the file exists in your code:
System.out.println("File exists : " + theXMLFile.length());
This is really only the file path.

You could try to load the file to make sure:

File file = new File(theXMLFile);

// Test if exists

// ...

// Load in ressource, I had issues loading confs and found this way is
working in all my cases, including custom FS in fs.default.name

conf.addResource(file.getAbsoluteFile().toURI().toURL());
(This is not really optimized but I believe it is ok for a JUnit setup
phase)

Julien

2013/1/17 Jay Vyas <[EMAIL PROTECTED]>

> no difference.
>
> This is really odd.
>
> The only way I can get it to work is by the following:
>
>
> /**
>  * A simple test harness for glusterfs
>  * @author jvyas
>  */
> public class TestMapReducerGlusterFS {
>
>
>     @Test
>     public void test() throws Exception{
>         Configuration conf = new Configuration(false);
>         System.out.println("before reload:"+conf);
>         //necessary?
>         conf.set("fs.default.name", ".......");
>         conf.set("fs.blah.mount", "/mnt/......");
>         conf.set("fs.blahserver", "192.168.1.36");
>         conf.set("iomod","Off");
>          ...
>     }
> }
>
> Thinking that this is a little known issue in the current hadoop
> parser/configurator? because very rarely do people write unit tests which
> utilize abnormal ( i.e. non-local or non-hdfs) file systems.
>
>
> On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <[EMAIL PROTECTED]> wrote:
>
>> Result any different if you call reloadConfiguration()?
>>
>> Thanks,
>>
>> Tom
>>
>> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
>> > Hi guys:
>> >
>> > I've finally extracted my problem of loading a special filesystem
>> > into a unit test.
>> >
>> > Below, clearly, Im creating a raw configuration and adding a single
>> resource
>> > to it (core-site.xml).
>> >
>> > Afterwards, i print to confirm that the file exists.
>> >
>> > Finally, I try to create a FileSystem from that file.  However, no file
>> > system is created :(.
>> >
>> > Any thoughts on why this core-site.xml file is being ignoreed?
>> > To note: I've also tried putting core-site.xml in the working
>> directory, and
>> > in a local conf/ directory.  This absolute path was a last effort to
>> really
>> > confirm that the local path wasn't somehow being mangled by the junit
>> > runtime.
>> >
>> > @Test
>> >     public void test() throws Exception{
>> >         Configuration conf = new Configuration(false);
>> >         System.out.println("b4:"+conf);
>> >         String
>> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
>> >         conf.addResource(theXMLFile);
>> >         System.out.println("File exists : " + theXMLFile.length());
>> >         System.out.println("after:"+conf);
>> >         System.out.println("file sytem " + conf.get("fs.default.name
>> "));
>> >         conf.writeXml(System.out);
>> >
>> >
>> >         System.out.println("Working directory = " +
>> > System.getProperty("user.dir"));
>> >         org.apache.hadoop.fs.FileSystem fs >> > org.apache.hadoop.fs.FileSystem.get(conf);
>> >         System.out.println("before test filesystem = " + fs);
>> >         //WordCount.main(new
>> > String[]{"resources/wordcount.txt","wordcountout.txt"});
>> >     }
>> >
>> > --
>> > Jay Vyas
>> > http://jayunit100.blogspot.com
>>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>