Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - HDFS / Federated HDFS - Doubts


Copy link to this message
-
Re: HDFS / Federated HDFS - Doubts
Steve Edison 2013-10-16, 16:22
I have couple of questions about HDFS federation:

Can I state different block store directories for each namespace on a
datanode ?
Can I have some datanodes dedicated to a particular namespace only ?

This seems quite interesting. Way to go !
On Tue, Oct 1, 2013 at 9:52 PM, Krishna Kumaar Natarajan
<[EMAIL PROTECTED]>wrote:

> Hi All,
>
> While trying to understand federated HDFS in detail I had few doubts and
> listing them down for your help.
>
>    1. In case of *HDFS(without HDFS federation)*, the metadata or the
>    data about the blocks belonging to the files in HDFS is maintained in the
>    main memory of the name node or it is stored on permanent storage of the
>    namenode and is brought in the main memory on demand basis ? [Krishna]
>    Based on my understanding, I assume the entire metadata is in main memory
>    which is an issue by itself. Please correct me if my understanding is wrong.
>    2. In case of* federated HDFS*, the metadata or the data about the
>    blocks belonging to files in a particular namespace is maintained in the
>    main memory of the namenode or it is stored on the permanent storage of the
>    namenode and is brought in the main memory on demand basis ?
>    3. Are the metadata information stored in separate cluster nodes(block
>    management layer separation) as discussed in Appendix B of this document ?
>    https://issues.apache.org/jira/secure/attachment/12453067/high-level-design.pdf
>    4. I would like to know if the following proposals are already
>    implemented in federated HDFS. (
>    http://www.slideshare.net/hortonworks/hdfs-futures-namenode-federation-for-improved-efficiency-and-scalability
>     slide-17)
>    - Separation of namespace and block management layers (same as qn.3)
>       - Partial namespace in memory for further scalability
>       - Move partial namespace from one namenode to another
>
> Thanks,
> Krishna
>