-Re: HDFS / Federated HDFS - Doubts
Chris Mawata 2013-10-02, 18:43
One more thing, Krishna, when using JounalNodes as opposed to the
native file system for the metadata storage you do get replication.
On 10/2/2013 12:52 AM, Krishna Kumaar Natarajan wrote:
> Hi All,
> While trying to understand federated HDFS in detail I had few doubts
> and listing them down for your help.
> 1. In case of *_HDFS(without HDFS federation)_*, the metadata or the
> data about the blocks belonging to the files in HDFS is maintained
> in the main memory of the name node or it is stored on permanent
> storage of the namenode and is brought in the main memory on
> demand basis ?[Krishna] Based on my understanding, I assume the
> entire metadata is in main memory which is an issue by itself.
> Please correct me if my understanding is wrong.
> 2. In case of*_federated HDFS_*, the metadata or the data about the
> blocks belonging to files in a particular namespace is maintained
> in the main memory of the namenode or it is stored on the
> permanent storage of the namenode and is brought in the main
> memory on demand basis ?
> 3. Are the metadata information stored in separate cluster
> nodes(block management layer separation) as discussed in Appendix
> B of this document
> 4. I would like to know if the following proposals are already
> implemented in federated HDFS.
> (http://www.slideshare.net/hortonworks/hdfs-futures-namenode-federation-for-improved-efficiency-and-scalability slide-17)
> * Separation of namespace and block management layers (same as qn.3)
> * Partial namespace in memory for further scalability
> * Move partial namespace from one namenode to another