Hi,

I am trying to add a file to HDFS programmatically.

In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration object as follows

Configuration configuration = null;
configuration.addResource(new URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
configuration.addResource(new Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
configuration.reloadConfiguration();
fileSystem = FileSystem.get(configuration);
System.out.println(fileSystem.getName());

This code prints file system as file:/// instead of hdfs://.

When I manually set HDFS configuration parameter like below, it works well

//configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");

How can I make my code work? Any inputs would be greatly appreciated.

Regards,
Anand.C

Hello there,

        Add the "conf/core-site.xml" file as well.

Regards,
    Mohammad Tariq
On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
<[EMAIL PROTECTED]> wrote:
> Hi,
>
>
>
> I am trying to add a file to HDFS programmatically.
>
>
>
> In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
> object as follows
>
>
>
> Configuration configuration = null;
>
> configuration.addResource(new
> URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>
> configuration.addResource(new
> Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>
> configuration.reloadConfiguration();
>
> fileSystem = FileSystem.get(configuration);
>
> System.out.println(fileSystem.getName());
>
>
>
> This code prints file system as file:/// instead of hdfs://.
>
>
>
> When I manually set HDFS configuration parameter like below, it works well
>
>
>
> //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");
>
>
>
> How can I make my code work? Any inputs would be greatly appreciated.
>
>
>
> Regards,
>
> Anand.C
>
>
>
>

Hi,

I have added other xml files too. But the issue is configuration object is not getting updated with the xml conents.

When I add

System.out.println(configuration.get("fs.default.name"));

I am not getting my hdfs url which I have in core-site.xml. When I run my code in eclipse, it works. But when I make it as a runnable jar file, it outputs file:///

Regards,
Anand.C

-----Original Message-----
From: Mohammad Tariq [mailto:[EMAIL PROTECTED]]
Sent: Thursday, August 09, 2012 1:28 PM
To: [EMAIL PROTECTED]
Subject: Re: Question on adding Hadoop XML to Configuration Object

Hello there,

        Add the "conf/core-site.xml" file as well.

Regards,
    Mohammad Tariq
On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
<[EMAIL PROTECTED]> wrote:
> Hi,
>
>
>
> I am trying to add a file to HDFS programmatically.
>
>
>
> In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
> object as follows
>
>
>
> Configuration configuration = null;
>
> configuration.addResource(new
> URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>
> configuration.addResource(new
> Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>
> configuration.reloadConfiguration();
>
> fileSystem = FileSystem.get(configuration);
>
> System.out.println(fileSystem.getName());
>
>
>
> This code prints file system as file:/// instead of hdfs://.
>
>
>
> When I manually set HDFS configuration parameter like below, it works well
>
>
>
> //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");
>
>
>
> How can I make my code work? Any inputs would be greatly appreciated.
>
>
>
> Regards,
>
> Anand.C
>
>
>
>

Try this and let me know if it works,

                Configuration conf = new Configuration();
conf.addResource(new Path("YOUR_HADOOP_HOME/conf/core-site.xml"));
conf.addResource(new Path("YOUR_HADOOP_HOME/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(conf);
Regards,
    Mohammad Tariq
On Thu, Aug 9, 2012 at 1:58 PM, Chandra Mohan, Ananda Vel Murugan
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have added other xml files too. But the issue is configuration object is not getting updated with the xml conents.
>
> When I add
>
>         System.out.println(configuration.get("fs.default.name"));
>
> I am not getting my hdfs url which I have in core-site.xml. When I run my code in eclipse, it works. But when I make it as a runnable jar file, it outputs file:///
>
> Regards,
> Anand.C
>
> -----Original Message-----
> From: Mohammad Tariq [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 09, 2012 1:28 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Question on adding Hadoop XML to Configuration Object
>
> Hello there,
>
>         Add the "conf/core-site.xml" file as well.
>
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
> <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>>
>>
>> I am trying to add a file to HDFS programmatically.
>>
>>
>>
>> In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
>> object as follows
>>
>>
>>
>> Configuration configuration = null;
>>
>> configuration.addResource(new
>> URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>>
>> configuration.addResource(new
>> Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>>
>> configuration.reloadConfiguration();
>>
>> fileSystem = FileSystem.get(configuration);
>>
>> System.out.println(fileSystem.getName());
>>
>>
>>
>> This code prints file system as file:/// instead of hdfs://.
>>
>>
>>
>> When I manually set HDFS configuration parameter like below, it works well
>>
>>
>>
>> //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");
>>
>>
>>
>> How can I make my code work? Any inputs would be greatly appreciated.
>>
>>
>>
>> Regards,
>>
>> Anand.C
>>
>>
>>
>>

Instead of setting the xml files programmatically, why not set the
HADOOP_CONF_DIR env variable to '/usr/local/hadoop-1.0.2/conf/'?
That way, you can just create a new Configuration() object and the files
will be loaded for you without any extra work.

On 9 August 2012 09:28, Chandra Mohan, Ananda Vel Murugan <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have added other xml files too. But the issue is configuration object is
> not getting updated with the xml conents.
>
> When I add
>
>         System.out.println(configuration.get("fs.default.name"));
>
> I am not getting my hdfs url which I have in core-site.xml. When I run my
> code in eclipse, it works. But when I make it as a runnable jar file, it
> outputs file:///
>
> Regards,
> Anand.C
>
> -----Original Message-----
> From: Mohammad Tariq [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 09, 2012 1:28 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Question on adding Hadoop XML to Configuration Object
>
> Hello there,
>
>         Add the "conf/core-site.xml" file as well.
>
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> >
> >
> > I am trying to add a file to HDFS programmatically.
> >
> >
> >
> > In my code, I am adding hdfs-site.xml and other xml to Hadoop
> Configuration
> > object as follows
> >
> >
> >
> > Configuration configuration = null;
> >
> > configuration.addResource(new
> > URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
> >
> > configuration.addResource(new
> > Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
> >
> > configuration.reloadConfiguration();
> >
> > fileSystem = FileSystem.get(configuration);
> >
> > System.out.println(fileSystem.getName());
> >
> >
> >
> > This code prints file system as file:/// instead of hdfs://.
> >
> >
> >
> > When I manually set HDFS configuration parameter like below, it works
> well
> >
> >
> >
> > //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");
> >
> >
> >
> > How can I make my code work? Any inputs would be greatly appreciated.
> >
> >
> >
> > Regards,
> >
> > Anand.C
> >
> >
> >
> >
>

Also - make sure the fs.default.name parameter is actually in
core-site.xml, not hdfs-site.xml

On 9 August 2012 09:32, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Try this and let me know if it works,
>
>                 Configuration conf = new Configuration();
>                 conf.addResource(new
> Path("YOUR_HADOOP_HOME/conf/core-site.xml"));
>                 conf.addResource(new
> Path("YOUR_HADOOP_HOME/conf/hdfs-site.xml"));
>                 FileSystem fs = FileSystem.get(conf);
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Aug 9, 2012 at 1:58 PM, Chandra Mohan, Ananda Vel Murugan
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I have added other xml files too. But the issue is configuration object
> is not getting updated with the xml conents.
> >
> > When I add
> >
> >         System.out.println(configuration.get("fs.default.name"));
> >
> > I am not getting my hdfs url which I have in core-site.xml. When I run
> my code in eclipse, it works. But when I make it as a runnable jar file, it
> outputs file:///
> >
> > Regards,
> > Anand.C
> >
> > -----Original Message-----
> > From: Mohammad Tariq [mailto:[EMAIL PROTECTED]]
> > Sent: Thursday, August 09, 2012 1:28 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Question on adding Hadoop XML to Configuration Object
> >
> > Hello there,
> >
> >         Add the "conf/core-site.xml" file as well.
> >
> > Regards,
> >     Mohammad Tariq
> >
> >
> > On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
> > <[EMAIL PROTECTED]> wrote:
> >> Hi,
> >>
> >>
> >>
> >> I am trying to add a file to HDFS programmatically.
> >>
> >>
> >>
> >> In my code, I am adding hdfs-site.xml and other xml to Hadoop
> Configuration
> >> object as follows
> >>
> >>
> >>
> >> Configuration configuration = null;
> >>
> >> configuration.addResource(new
> >> URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
> >>
> >> configuration.addResource(new
> >> Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
> >>
> >> configuration.reloadConfiguration();
> >>
> >> fileSystem = FileSystem.get(configuration);
> >>
> >> System.out.println(fileSystem.getName());
> >>
> >>
> >>
> >> This code prints file system as file:/// instead of hdfs://.
> >>
> >>
> >>
> >> When I manually set HDFS configuration parameter like below, it works
> well
> >>
> >>
> >>
> >> //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");
> >>
> >>
> >>
> >> How can I make my code work? Any inputs would be greatly appreciated.
> >>
> >>
> >>
> >> Regards,
> >>
> >> Anand.C
> >>
> >>
> >>
> >>
>

Also, Hadoop jars must be in the classpath when you run the app from
command line or as a jar.

Regards,
    Mohammad Tariq
On Thu, Aug 9, 2012 at 2:02 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Try this and let me know if it works,
>
>                 Configuration conf = new Configuration();
>                 conf.addResource(new Path("YOUR_HADOOP_HOME/conf/core-site.xml"));
>                 conf.addResource(new Path("YOUR_HADOOP_HOME/conf/hdfs-site.xml"));
>                 FileSystem fs = FileSystem.get(conf);
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Aug 9, 2012 at 1:58 PM, Chandra Mohan, Ananda Vel Murugan
> <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> I have added other xml files too. But the issue is configuration object is not getting updated with the xml conents.
>>
>> When I add
>>
>>         System.out.println(configuration.get("fs.default.name"));
>>
>> I am not getting my hdfs url which I have in core-site.xml. When I run my code in eclipse, it works. But when I make it as a runnable jar file, it outputs file:///
>>
>> Regards,
>> Anand.C
>>
>> -----Original Message-----
>> From: Mohammad Tariq [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, August 09, 2012 1:28 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Question on adding Hadoop XML to Configuration Object
>>
>> Hello there,
>>
>>         Add the "conf/core-site.xml" file as well.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
>> <[EMAIL PROTECTED]> wrote:
>>> Hi,
>>>
>>>
>>>
>>> I am trying to add a file to HDFS programmatically.
>>>
>>>
>>>
>>> In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
>>> object as follows
>>>
>>>
>>>
>>> Configuration configuration = null;
>>>
>>> configuration.addResource(new
>>> URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>>>
>>> configuration.addResource(new
>>> Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>>>
>>> configuration.reloadConfiguration();
>>>
>>> fileSystem = FileSystem.get(configuration);
>>>
>>> System.out.println(fileSystem.getName());
>>>
>>>
>>>
>>> This code prints file system as file:/// instead of hdfs://.
>>>
>>>
>>>
>>> When I manually set HDFS configuration parameter like below, it works well
>>>
>>>
>>>
>>> //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");
>>>
>>>
>>>
>>> How can I make my code work? Any inputs would be greatly appreciated.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Anand.C
>>>
>>>
>>>
>>>

Thanks all. I had a typo in core-site.xml path. It works now.

It would be better if conf.addResource throws an exception like FileNotFoundException when it is not able to find the xml file specified.

I like idea of setting HADOOP_CONF_DIR. I will try that too.

Regards,
Anand.C

-----Original Message-----
From: Mohammad Tariq [mailto:[EMAIL PROTECTED]]
Sent: Thursday, August 09, 2012 2:04 PM
To: [EMAIL PROTECTED]
Subject: Re: Question on adding Hadoop XML to Configuration Object

Also, Hadoop jars must be in the classpath when you run the app from
command line or as a jar.

Regards,
    Mohammad Tariq
On Thu, Aug 9, 2012 at 2:02 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Try this and let me know if it works,
>
>                 Configuration conf = new Configuration();
>                 conf.addResource(new Path("YOUR_HADOOP_HOME/conf/core-site.xml"));
>                 conf.addResource(new Path("YOUR_HADOOP_HOME/conf/hdfs-site.xml"));
>                 FileSystem fs = FileSystem.get(conf);
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Aug 9, 2012 at 1:58 PM, Chandra Mohan, Ananda Vel Murugan
> <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> I have added other xml files too. But the issue is configuration object is not getting updated with the xml conents.
>>
>> When I add
>>
>>         System.out.println(configuration.get("fs.default.name"));
>>
>> I am not getting my hdfs url which I have in core-site.xml. When I run my code in eclipse, it works. But when I make it as a runnable jar file, it outputs file:///
>>
>> Regards,
>> Anand.C
>>
>> -----Original Message-----
>> From: Mohammad Tariq [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, August 09, 2012 1:28 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Question on adding Hadoop XML to Configuration Object
>>
>> Hello there,
>>
>>         Add the "conf/core-site.xml" file as well.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
>> <[EMAIL PROTECTED]> wrote:
>>> Hi,
>>>
>>>
>>>
>>> I am trying to add a file to HDFS programmatically.
>>>
>>>
>>>
>>> In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
>>> object as follows
>>>
>>>
>>>
>>> Configuration configuration = null;
>>>
>>> configuration.addResource(new
>>> URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>>>
>>> configuration.addResource(new
>>> Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
>>>
>>> configuration.reloadConfiguration();
>>>
>>> fileSystem = FileSystem.get(configuration);
>>>
>>> System.out.println(fileSystem.getName());
>>>
>>>
>>>
>>> This code prints file system as file:/// instead of hdfs://.
>>>
>>>
>>>
>>> When I manually set HDFS configuration parameter like below, it works well
>>>
>>>
>>>
>>> //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");
>>>
>>>
>>>
>>> How can I make my code work? Any inputs would be greatly appreciated.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Anand.C
>>>
>>>
>>>
>>>