Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> [Proposal] Pluggable Namespace


Copy link to this message
-
Re: [Proposal] Pluggable Namespace
I think it would be awesome to make this happen.  I am +1 for the general
direction. I am by no means an HDFS expert but I would like to see the
patches you have made.  If you could file a JIRA that points to the
modifications on github or in patches and any design work you have
explaining it (including what you have below) that would give us a place
to review it and discuss it.

--Bobby

On 10/3/13 2:17 PM, "Milind Bhandarkar" <[EMAIL PROTECTED]> wrote:

>Hi All,
>
>Exec Summary: For the last couple of months, we, at Pivotal, along with a
>couple of folks in the community have been working on making Namespace
>implementation in the namenode pluggable. We have demonstrated that it can
>be done without major surgery on the namenode, and does not have
>noticeable
>performance impact. We would like to contribute it back to Apache if there
>is sufficient interest. Please let us know if you are interested, and we
>will create a Jira and update the patch for in-progress work.
>
>
>Rationale:
>
>In a Hadoop cluster, Namenode roughly has following main responsibilities.
>€ Catering to RPC calls from clients.
>€ Managing the HDFS namespace tree.
>€ Managing block report, heartbeat and other communication from data
>nodes.
>
>For Hadoop clusters having large number of files and large number of
>nodes,
>name node gets bottlenecked. Mainly for two reasons
>€ All the information is kept in name node¹s main memory.
>€ Namenode has to cater to all the request from clients / data nodes.
>€ And also perform some operations for backup and check pointing node.
>
>A possible solution is to add more main memory but there are certain
>issues
>with this approach
>€ Namnenode being Java application, garbage collection cycles execute
>periodically to reclaim unreferenced heap space. When the heap space grows
>very large, despite of GC policy  chosen, application stalls during the GC
>activity. This creates a bunch of issues since DNs and  clients may
>perceive this stall as NN crash.
>€ There will always be a practical limit on how much physical memory a
>single machine can accommodate.
>
>Proposed Solution:
>
>Out of the three responsibilities listed above, we can refactor namespace
>management from the namenode codebase in such a way that there is
>provision
>to implement and plug other name systems other than existing in-process
>memory-based name system. Particularly a name system backed by a
>distributed key-value store will significantly reduce namenode memory
>requirement.To achieve this, a new generic interface will be introduced
>[Let¹s call it AbstractNameSystem] which defines set of operations using
>which we perform the namespace management. Namenode code that used to
>manipulate some java objects maintained in namenode¹s heap will now
>operate
>on this interface. There will be provision for others to extend this
>interface and plug their own NameSystem implementation.
>
>To get started, we have implemented the same memory-based namespace
>implementation in a remote process, outside of the namenode JVM. In
>addition, work is undergoing to implement the namesystem using HBase.
>
>Details of Changes:
>
>Created new class called AbstractNamesystem, existing FSNamesystem is a
>subclass of this class. Some code from FSNamesystem has been moved to its
>parent. Created a Factory class to create object of NS management
>class.Factory refers to newly added config properties to support pluggable
>name space management class. Added unit tests for Factory. Replaced
>constructors with factory calls, this is  because the namesystem instances
>should now be created based on configuration. Added new config properties
>to support pluggable name space management class. This property will
>decide
>which Namesystem class will be instantiated by the factory. This change is
>also reflected in some DFS related webapps [JSP files] where namesystem
>instance is used to obtain DFS health and other stats.
>
>These changes aim to make the namesystem pluggable without changing high
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB