Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - [Proposal] Pluggable Namespace


Copy link to this message
-
Re: [Proposal] Pluggable Namespace
Bobby Evans 2013-10-04, 15:31
I think it would be awesome to make this happen.  I am +1 for the general
direction. I am by no means an HDFS expert but I would like to see the
patches you have made.  If you could file a JIRA that points to the
modifications on github or in patches and any design work you have
explaining it (including what you have below) that would give us a place
to review it and discuss it.

--Bobby

On 10/3/13 2:17 PM, "Milind Bhandarkar" <[EMAIL PROTECTED]> wrote:

>Hi All,
>
>Exec Summary: For the last couple of months, we, at Pivotal, along with a
>couple of folks in the community have been working on making Namespace
>implementation in the namenode pluggable. We have demonstrated that it can
>be done without major surgery on the namenode, and does not have
>noticeable
>performance impact. We would like to contribute it back to Apache if there
>is sufficient interest. Please let us know if you are interested, and we
>will create a Jira and update the patch for in-progress work.
>
>
>Rationale:
>
>In a Hadoop cluster, Namenode roughly has following main responsibilities.
>€ Catering to RPC calls from clients.
>€ Managing the HDFS namespace tree.
>€ Managing block report, heartbeat and other communication from data
>nodes.
>
>For Hadoop clusters having large number of files and large number of
>nodes,
>name node gets bottlenecked. Mainly for two reasons
>€ All the information is kept in name node¹s main memory.
>€ Namenode has to cater to all the request from clients / data nodes.
>€ And also perform some operations for backup and check pointing node.
>
>A possible solution is to add more main memory but there are certain
>issues
>with this approach
>€ Namnenode being Java application, garbage collection cycles execute
>periodically to reclaim unreferenced heap space. When the heap space grows
>very large, despite of GC policy  chosen, application stalls during the GC
>activity. This creates a bunch of issues since DNs and  clients may
>perceive this stall as NN crash.
>€ There will always be a practical limit on how much physical memory a
>single machine can accommodate.
>
>Proposed Solution:
>
>Out of the three responsibilities listed above, we can refactor namespace
>management from the namenode codebase in such a way that there is
>provision
>to implement and plug other name systems other than existing in-process
>memory-based name system. Particularly a name system backed by a
>distributed key-value store will significantly reduce namenode memory
>requirement.To achieve this, a new generic interface will be introduced
>[Let¹s call it AbstractNameSystem] which defines set of operations using
>which we perform the namespace management. Namenode code that used to
>manipulate some java objects maintained in namenode¹s heap will now
>operate
>on this interface. There will be provision for others to extend this
>interface and plug their own NameSystem implementation.
>
>To get started, we have implemented the same memory-based namespace
>implementation in a remote process, outside of the namenode JVM. In
>addition, work is undergoing to implement the namesystem using HBase.
>
>Details of Changes:
>
>Created new class called AbstractNamesystem, existing FSNamesystem is a
>subclass of this class. Some code from FSNamesystem has been moved to its
>parent. Created a Factory class to create object of NS management
>class.Factory refers to newly added config properties to support pluggable
>name space management class. Added unit tests for Factory. Replaced
>constructors with factory calls, this is  because the namesystem instances
>should now be created based on configuration. Added new config properties
>to support pluggable name space management class. This property will
>decide
>which Namesystem class will be instantiated by the factory. This change is
>also reflected in some DFS related webapps [JSP files] where namesystem
>instance is used to obtain DFS health and other stats.
>
>These changes aim to make the namesystem pluggable without changing high