Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - [Proposal] Pluggable Namespace


Copy link to this message
-
[Proposal] Pluggable Namespace
Milind Bhandarkar 2013-10-03, 19:17
Hi All,

Exec Summary: For the last couple of months, we, at Pivotal, along with a
couple of folks in the community have been working on making Namespace
implementation in the namenode pluggable. We have demonstrated that it can
be done without major surgery on the namenode, and does not have noticeable
performance impact. We would like to contribute it back to Apache if there
is sufficient interest. Please let us know if you are interested, and we
will create a Jira and update the patch for in-progress work.
Rationale:

In a Hadoop cluster, Namenode roughly has following main responsibilities.
• Catering to RPC calls from clients.
• Managing the HDFS namespace tree.
• Managing block report, heartbeat and other communication from data nodes.

For Hadoop clusters having large number of files and large number of nodes,
name node gets bottlenecked. Mainly for two reasons
• All the information is kept in name node’s main memory.
• Namenode has to cater to all the request from clients / data nodes.
• And also perform some operations for backup and check pointing node.

A possible solution is to add more main memory but there are certain issues
with this approach
• Namnenode being Java application, garbage collection cycles execute
periodically to reclaim unreferenced heap space. When the heap space grows
very large, despite of GC policy  chosen, application stalls during the GC
activity. This creates a bunch of issues since DNs and  clients may
perceive this stall as NN crash.
• There will always be a practical limit on how much physical memory a
single machine can accommodate.

Proposed Solution:

Out of the three responsibilities listed above, we can refactor namespace
management from the namenode codebase in such a way that there is provision
to implement and plug other name systems other than existing in-process
memory-based name system. Particularly a name system backed by a
distributed key-value store will significantly reduce namenode memory
requirement.To achieve this, a new generic interface will be introduced
[Let’s call it AbstractNameSystem] which defines set of operations using
which we perform the namespace management. Namenode code that used to
manipulate some java objects maintained in namenode’s heap will now operate
on this interface. There will be provision for others to extend this
interface and plug their own NameSystem implementation.

To get started, we have implemented the same memory-based namespace
implementation in a remote process, outside of the namenode JVM. In
addition, work is undergoing to implement the namesystem using HBase.

Details of Changes:

Created new class called AbstractNamesystem, existing FSNamesystem is a
subclass of this class. Some code from FSNamesystem has been moved to its
parent. Created a Factory class to create object of NS management
class.Factory refers to newly added config properties to support pluggable
name space management class. Added unit tests for Factory. Replaced
constructors with factory calls, this is  because the namesystem instances
should now be created based on configuration. Added new config properties
to support pluggable name space management class. This property will decide
which Namesystem class will be instantiated by the factory. This change is
also reflected in some DFS related webapps [JSP files] where namesystem
instance is used to obtain DFS health and other stats.

These changes aim to make the namesystem pluggable without changing high
level interfaces, this is particularly tricky since memory-based name
system functionality is currently baked into these interfaces, and ultimate
goal is to make the high level interface free from memory-based name system.

Consideration for Upgrade and Rollback:

Current memory based implementation already has code to read from and write
to fsimage , we will have to make them publicly accessible which will
enable us to upgrade an existing cluster from FSNamespace to newly added
name system in future version.

a. Upgrades: By making use of existing Loader class for reading fsimage we
can write some code load this image into the future name system
implementation.

b. Rollback: Are even simpler, we can preserve the old fsimage and start
the cluster with that image by configuring the cluster to use current file
system based name system.

Future work

Current HDFS design is such that FSNameSystem is baked into even high level
interfaces, this is a major hurdle in cleanly implementing pluggable name
systems. We aim to propose a change in such interfaces into which
FSNameSystem is tightly coupled.

- Milind
Milind Bhandarkar
Chief Scientist
Pivotal
+
Bobby Evans 2013-10-04, 15:31
+
Andrew Purtell 2013-10-05, 18:23
+
Milind Bhandarkar 2013-10-05, 19:37
+
Azuryy Yu 2013-10-06, 01:40
+
Milind Bhandarkar 2013-10-06, 16:35
+
Vinod Kumar Vavilapalli 2013-10-06, 19:20
+
Milind Bhandarkar 2013-10-06, 19:54
+
Milind Bhandarkar 2013-10-07, 00:58
+
Mahadev Konar 2013-10-07, 03:04
+
Bobby Evans 2013-10-07, 15:52
+
Milind Bhandarkar 2013-10-07, 17:18
+
Chris Nauroth 2013-10-07, 16:44
+
Milind Bhandarkar 2013-10-07, 17:11
+
Doug Cutting 2013-10-07, 18:15
+
Andrew Purtell 2013-10-07, 19:05
+
Vinod Kumar Vavilapalli 2013-10-07, 19:42
+
Milind Bhandarkar 2013-10-07, 19:49
+
Doug Cutting 2013-10-07, 19:50
+
Milind Bhandarkar 2013-10-07, 23:50
+
Konstantin Shvachko 2013-10-08, 02:40
+
sanjay Radia 2013-10-07, 19:31
+
sanjay Radia 2013-10-08, 04:50
+
Milind Bhandarkar 2013-10-08, 17:13