Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - [Proposal] Pluggable Namespace


Copy link to this message
-
Re: [Proposal] Pluggable Namespace
Azuryy Yu 2013-10-06, 01:40
Hi Milind,

HDFS federation can solve the NN bottle neck and memory limit problem.

AbstractNameSystem design sounds good. but distributed meta storage using
HBase should bring performance degration.
On Oct 4, 2013 3:18 AM, "Milind Bhandarkar" <[EMAIL PROTECTED]>
wrote:

> Hi All,
>
> Exec Summary: For the last couple of months, we, at Pivotal, along with a
> couple of folks in the community have been working on making Namespace
> implementation in the namenode pluggable. We have demonstrated that it can
> be done without major surgery on the namenode, and does not have noticeable
> performance impact. We would like to contribute it back to Apache if there
> is sufficient interest. Please let us know if you are interested, and we
> will create a Jira and update the patch for in-progress work.
>
>
> Rationale:
>
> In a Hadoop cluster, Namenode roughly has following main responsibilities.
> • Catering to RPC calls from clients.
> • Managing the HDFS namespace tree.
> • Managing block report, heartbeat and other communication from data nodes.
>
> For Hadoop clusters having large number of files and large number of nodes,
> name node gets bottlenecked. Mainly for two reasons
> • All the information is kept in name node’s main memory.
> • Namenode has to cater to all the request from clients / data nodes.
> • And also perform some operations for backup and check pointing node.
>
> A possible solution is to add more main memory but there are certain issues
> with this approach
> • Namnenode being Java application, garbage collection cycles execute
> periodically to reclaim unreferenced heap space. When the heap space grows
> very large, despite of GC policy  chosen, application stalls during the GC
> activity. This creates a bunch of issues since DNs and  clients may
> perceive this stall as NN crash.
> • There will always be a practical limit on how much physical memory a
> single machine can accommodate.
>
> Proposed Solution:
>
> Out of the three responsibilities listed above, we can refactor namespace
> management from the namenode codebase in such a way that there is provision
> to implement and plug other name systems other than existing in-process
> memory-based name system. Particularly a name system backed by a
> distributed key-value store will significantly reduce namenode memory
> requirement.To achieve this, a new generic interface will be introduced
> [Let’s call it AbstractNameSystem] which defines set of operations using
> which we perform the namespace management. Namenode code that used to
> manipulate some java objects maintained in namenode’s heap will now operate
> on this interface. There will be provision for others to extend this
> interface and plug their own NameSystem implementation.
>
> To get started, we have implemented the same memory-based namespace
> implementation in a remote process, outside of the namenode JVM. In
> addition, work is undergoing to implement the namesystem using HBase.
>
> Details of Changes:
>
> Created new class called AbstractNamesystem, existing FSNamesystem is a
> subclass of this class. Some code from FSNamesystem has been moved to its
> parent. Created a Factory class to create object of NS management
> class.Factory refers to newly added config properties to support pluggable
> name space management class. Added unit tests for Factory. Replaced
> constructors with factory calls, this is  because the namesystem instances
> should now be created based on configuration. Added new config properties
> to support pluggable name space management class. This property will decide
> which Namesystem class will be instantiated by the factory. This change is
> also reflected in some DFS related webapps [JSP files] where namesystem
> instance is used to obtain DFS health and other stats.
>
> These changes aim to make the namesystem pluggable without changing high
> level interfaces, this is particularly tricky since memory-based name
> system functionality is currently baked into these interfaces, and ultimate