Result any different if you call reloadConfiguration()?

Thanks,

Tom

On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource
> to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.
>
> Finally, I try to create a FileSystem from that file.  However, no file
> system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and
> in a local conf/ directory.  This absolute path was a last effort to really
> confirm that the local path wasn't somehow being mangled by the junit
> runtime.
>
> @Test
>     public void test() throws Exception{
>         Configuration conf = new Configuration(false);
>         System.out.println("b4:"+conf);
>         String
> theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
>         conf.addResource(theXMLFile);
>         System.out.println("File exists : " + theXMLFile.length());
>         System.out.println("after:"+conf);
>         System.out.println("file sytem " + conf.get("fs.default.name"));
>         conf.writeXml(System.out);
>
>
>         System.out.println("Working directory = " +
> System.getProperty("user.dir"));
>         org.apache.hadoop.fs.FileSystem fs > org.apache.hadoop.fs.FileSystem.get(conf);
>         System.out.println("before test filesystem = " + fs);
>         //WordCount.main(new
> String[]{"resources/wordcount.txt","wordcountout.txt"});
>     }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com

no difference.

This is really odd.

The only way I can get it to work is by the following:
/**
 * A simple test harness for glusterfs
 * @author jvyas
 */
public class TestMapReducerGlusterFS {

    @Test
    public void test() throws Exception{
        Configuration conf = new Configuration(false);
        System.out.println("before reload:"+conf);
        //necessary?
        conf.set("fs.default.name", ".......");
        conf.set("fs.blah.mount", "/mnt/......");
        conf.set("fs.blahserver", "192.168.1.36");
        conf.set("iomod","Off");
         ...
    }
}

Thinking that this is a little known issue in the current hadoop
parser/configurator? because very rarely do people write unit tests which
utilize abnormal ( i.e. non-local or non-hdfs) file systems.
On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <[EMAIL PROTECTED]> wrote:

> Result any different if you call reloadConfiguration()?
>
> Thanks,
>
> Tom
>
> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> > Hi guys:
> >
> > I've finally extracted my problem of loading a special filesystem
> > into a unit test.
> >
> > Below, clearly, Im creating a raw configuration and adding a single
> resource
> > to it (core-site.xml).
> >
> > Afterwards, i print to confirm that the file exists.
> >
> > Finally, I try to create a FileSystem from that file.  However, no file
> > system is created :(.
> >
> > Any thoughts on why this core-site.xml file is being ignoreed?
> > To note: I've also tried putting core-site.xml in the working directory,
> and
> > in a local conf/ directory.  This absolute path was a last effort to
> really
> > confirm that the local path wasn't somehow being mangled by the junit
> > runtime.
> >
> > @Test
> >     public void test() throws Exception{
> >         Configuration conf = new Configuration(false);
> >         System.out.println("b4:"+conf);
> >         String
> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> >         conf.addResource(theXMLFile);
> >         System.out.println("File exists : " + theXMLFile.length());
> >         System.out.println("after:"+conf);
> >         System.out.println("file sytem " + conf.get("fs.default.name"));
> >         conf.writeXml(System.out);
> >
> >
> >         System.out.println("Working directory = " +
> > System.getProperty("user.dir"));
> >         org.apache.hadoop.fs.FileSystem fs > > org.apache.hadoop.fs.FileSystem.get(conf);
> >         System.out.println("before test filesystem = " + fs);
> >         //WordCount.main(new
> > String[]{"resources/wordcount.txt","wordcountout.txt"});
> >     }
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>

--
Jay Vyas
http://jayunit100.blogspot.com

imo, as long as the javadoc is clear enough, any behavior is ok.
What's written in current version is:
name - resource to be added, the classpath is examined for a file with that
name.

So nothing in this javadoc lets you believe that the file exists or will be
loaded. Also you could make sure your file is in the classpath (which is
the case for me).

Julien

2013/1/17 Jay Vyas <[EMAIL PROTECTED]>

> Good catch with that string.length() - you're right, that was a silly
> mistake. --- sorry - im not sure what i was thinking. it was a late night
> :)
>
> In any case, the same code with file.exists() fails...  i've validated
> that path many ways.
>
> On a broader note: Shouldn't the Configuration class  just throw a
> RuntimeException if the file isn't available?    Its very strange to add
> all these default resources, which may or may not exist, to a class - then
> having to introspect on the class at runtime to see if they actually got
> loaded.
>
>
> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <[EMAIL PROTECTED]>wrote:
>
>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>

The configuration XML files are loaded from your CLASSPATH.

If you want to know more about what configuration you're picking up, you
can use the "./bin/hdfs getconf" command.

cheers,
Colin
On Thu, Jan 17, 2013 at 7:49 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> I agree, but the fact that the configuration is doing the loading, means
> (i think) that it should (at least) do some error handling for that
> loading, correct?
>
>
> On Thu, Jan 17, 2013 at 10:36 AM, Julien Muller <[EMAIL PROTECTED]>wrote:
>
>> imo, as long as the javadoc is clear enough, any behavior is ok.
>> What's written in current version is:
>> name - resource to be added, the classpath is examined for a file with
>> that name.
>>
>> So nothing in this javadoc lets you believe that the file exists or will
>> be loaded. Also you could make sure your file is in the classpath (which is
>> the case for me).
>>
>> Julien
>>
>> 2013/1/17 Jay Vyas <[EMAIL PROTECTED]>
>>
>>> Good catch with that string.length() - you're right, that was a silly
>>> mistake. --- sorry - im not sure what i was thinking. it was a late night
>>> :)
>>>
>>> In any case, the same code with file.exists() fails...  i've validated
>>> that path many ways.
>>>
>>> On a broader note: Shouldn't the Configuration class  just throw a
>>> RuntimeException if the file isn't available?    Its very strange to add
>>> all these default resources, which may or may not exist, to a class - then
>>> having to introspect on the class at runtime to see if they actually got
>>> loaded.
>>>
>>>
>>> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <[EMAIL PROTECTED]>wrote:
>>>
>>>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Jay Vyas
>>> http://jayunit100.blogspot.com
>>>
>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>

Can you try prepending the path with file:// or pass in a URL and use the method which takes in a URL?

I remember class loading issues with paths without the file: scheme info from a different scenario (while playing with logj4 config files).

HTH,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Jan 16, 2013, at 4:33 PM, Jay Vyas wrote:

> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.  
>
> Finally, I try to create a FileSystem from that file.  However, no file system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and in a local conf/ directory.  This absolute path was a last effort to really confirm that the local path wasn't somehow being mangled by the junit runtime.
>
> @Test
>     public void test() throws Exception{
>         Configuration conf = new Configuration(false);
>         System.out.println("b4:"+conf);
>         String theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
>         conf.addResource(theXMLFile);
>         System.out.println("File exists : " + theXMLFile.length());
>         System.out.println("after:"+conf);
>         System.out.println("file sytem " + conf.get("fs.default.name"));
>         conf.writeXml(System.out);
>
>        
>         System.out.println("Working directory = " + System.getProperty("user.dir"));
>         org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
>         System.out.println("before test filesystem = " + fs);
>         //WordCount.main(new String[]{"resources/wordcount.txt","wordcountout.txt"});
>     }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com