Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - HDFS / Federated HDFS - Doubts


Copy link to this message
-
Re: HDFS / Federated HDFS - Doubts
Chris Mawata 2013-10-02, 18:43
One more thing, Krishna,  when using JounalNodes as opposed to the
native file system for the metadata storage  you do get replication.
Chris
On 10/2/2013 12:52 AM, Krishna Kumaar Natarajan wrote:
> Hi All,
>
> While trying to understand federated HDFS in detail I had few doubts
> and listing them down for your help.
>
>  1. In case of *_HDFS(without HDFS federation)_*, the metadata or the
>     data about the blocks belonging to the files in HDFS is maintained
>     in the main memory of the name node or it is stored on permanent
>     storage of the namenode and is brought in the main memory on
>     demand basis ?[Krishna] Based on my understanding, I assume the
>     entire metadata is in main memory which is an issue by itself.
>     Please correct me if my understanding is wrong.
>  2. In case of*_federated HDFS_*, the metadata or the data about the
>     blocks belonging to files in a particular namespace is maintained
>     in the main memory of the namenode or it is stored on the
>     permanent storage of the namenode and is brought in the main
>     memory on demand basis ?
>  3. Are the metadata information stored in separate cluster
>     nodes(block management layer separation) as discussed in Appendix
>     B of this document
>     ?https://issues.apache.org/jira/secure/attachment/12453067/high-level-design.pdf
>  4. I would like to know if the following proposals are already
>     implemented in federated HDFS.
>     (http://www.slideshare.net/hortonworks/hdfs-futures-namenode-federation-for-improved-efficiency-and-scalability slide-17)
>       * Separation of namespace and block management layers (same as qn.3)
>       * Partial namespace in memory for further scalability
>       * Move partial namespace from one namenode to another
>
> Thanks,
> Krishna