Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Setting up Federated HDFS


Copy link to this message
-
Re: Setting up Federated HDFS
Praveen Sripati 2012-02-09, 14:22
Chandra,

In the namenode hdfs*xml, dfs.federation.nameservice.id is set to ns1, but
ns1 is not being used in the xml for defining the name node properties..

Here are the instructions to getting started with HDFS federation and mount
tables.

http://www.thecloudavenue.com/2012/01/getting-started-with-hdfs-federation.html

http://www.thecloudavenue.com/2012/01/hdfs-client-side-mount-table.html

Praveen

On Thu, Feb 9, 2012 at 2:12 PM, Chandrasekar <[EMAIL PROTECTED]>wrote:

> Hello,
>           I'm still having problems configuring a cluster running
> federated HDFS, in a single node. Here's what i've done so far:
>
>         1. Extracted hdfs 0.23 tarball at 4 different locations, one for
> each of the 3 NameNode daemons and 1 for DataNode and client
>
>         2. I've added hdfs-site.xml and core-site.xml having the following
> information to each of the NameNode:
>
>                             core-site.xml ----
> http://pastebin.com/nJaTq00s
>                             hdfs-site.xml ----
> http://pastebin.com/DkueDHDf
>
>             I've specified different port numbers, http port numbers and
> nameservice ids for each NameNode
>
>         3. Finally, to the fourth copy (DataNode) i've added the following
> configuration information:
>
>                             core-site.xml -----
> http://pastebin.com/DYfwfi44 <http://pastebin.com/58FJfTnn>
>                             hdfs-site.xml -----
> http://pastebin.com/xnmNnCKq
>
>             I've specified fs.defaultFs here as I'll be using this as my
> client too.
>
>
>         4. I then go to each bin folder and run the "hadoop namenode
> -format" and "hadoop namenode"
>             I haven't set the HADOOP_HOME env variable yet. (all namenodes
> run without any exception)
>
>         5. I then go to the DataNode bin folder and run "hadoop datanode".
> (no exception here)
>
>         6. Then I set the HADOOP_HOME env variable to the hadoop
> distribution from which the ran the DataNode.
>
>         7. Finally, I try to list out the contents of the viewfs file
> system by running "*hadoop fs -ls /*"
>
>             The output i'm getting is only this : "*ls:
> viewfs://localhost:33332/*"
>
>   Is this the right way to configure federated HDFS?
>
>
>
> On Wed, Feb 8, 2012 at 12:28, Suresh Srinivas <[EMAIL PROTECTED]>wrote:
>
>>
>>
>> On Tue, Feb 7, 2012 at 4:51 PM, Chandrasekar <[EMAIL PROTECTED]>wrote:
>>
>>>       In which file should i specify all this information about
>>> nameservices and the list of namenodes?
>>>
>>
>> hdfs-site.xml is the appropriate place, since it is hdfs-specific
>> configuration.
>>
>>        If there are multiple namenodes, then which one should i specify
>>> in core-site.xml as fs.defaultFS?
>>>
>>
>> core-site.xml is the right place for fs.defaultFS.
>>
>> Given you have multiple namespaces from in federation setup, fs.defaultFS
>> should point to ViewFileSystem for a unified view of the namespaces to the
>> clients. There is an open bug HDFS-2558 to track this. I will get to this
>> as soon as I can.
>>
>> Regards,
>> Suresh
>>
>
>