-Re: HDFS / Federated HDFS - Doubts
Steve Edison 2013-10-16, 16:22
I have couple of questions about HDFS federation:
Can I state different block store directories for each namespace on a
Can I have some datanodes dedicated to a particular namespace only ?
This seems quite interesting. Way to go !
On Tue, Oct 1, 2013 at 9:52 PM, Krishna Kumaar Natarajan
> Hi All,
> While trying to understand federated HDFS in detail I had few doubts and
> listing them down for your help.
> 1. In case of *HDFS(without HDFS federation)*, the metadata or the
> data about the blocks belonging to the files in HDFS is maintained in the
> main memory of the name node or it is stored on permanent storage of the
> namenode and is brought in the main memory on demand basis ? [Krishna]
> Based on my understanding, I assume the entire metadata is in main memory
> which is an issue by itself. Please correct me if my understanding is wrong.
> 2. In case of* federated HDFS*, the metadata or the data about the
> blocks belonging to files in a particular namespace is maintained in the
> main memory of the namenode or it is stored on the permanent storage of the
> namenode and is brought in the main memory on demand basis ?
> 3. Are the metadata information stored in separate cluster nodes(block
> management layer separation) as discussed in Appendix B of this document ?
> 4. I would like to know if the following proposals are already
> implemented in federated HDFS. (
> - Separation of namespace and block management layers (same as qn.3)
> - Partial namespace in memory for further scalability
> - Move partial namespace from one namenode to another