Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> [Proposal] Pluggable Namespace

Copy link to this message
Re: [Proposal] Pluggable Namespace
In order to make federation happen, the block pool management was already separated. Isn't that the same as this effortt?


On Oct 6, 2013, at 9:35 AM, Milind Bhandarkar wrote:

> Federation is orthogonal with Pluggable Namespaces. That is, one can use
> Federation if needed, even while a distributed K-V store is used on the
> backend.
> Limitations of Federated namenode for scaling namespace are
> well-documented in several places, including the Giraffa presentation.
> HBase is only one of the several namespace implementations possible. Thus,
> if HBase-based namespace implementation does not fit your performance
> needs, you have a choice of using something else.
> - milind
> -----Original Message-----
> From: Azuryy Yu [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, October 05, 2013 6:41 PM
> Subject: Re: [Proposal] Pluggable Namespace
> Hi Milind,
> HDFS federation can solve the NN bottle neck and memory limit problem.
> AbstractNameSystem design sounds good. but distributed meta storage using
> HBase should bring performance degration.
> On Oct 4, 2013 3:18 AM, "Milind Bhandarkar" <[EMAIL PROTECTED]>
> wrote:
>> Hi All,
>> Exec Summary: For the last couple of months, we, at Pivotal, along
>> with a couple of folks in the community have been working on making
>> Namespace implementation in the namenode pluggable. We have
>> demonstrated that it can be done without major surgery on the
>> namenode, and does not have noticeable performance impact. We would
>> like to contribute it back to Apache if there is sufficient interest.
>> Please let us know if you are interested, and we will create a Jira and
> update the patch for in-progress work.
>> Rationale:
>> In a Hadoop cluster, Namenode roughly has following main
> responsibilities.
>> . Catering to RPC calls from clients.
>> . Managing the HDFS namespace tree.
>> . Managing block report, heartbeat and other communication from data
> nodes.
>> For Hadoop clusters having large number of files and large number of
>> nodes, name node gets bottlenecked. Mainly for two reasons . All the
>> information is kept in name node's main memory.
>> . Namenode has to cater to all the request from clients / data nodes.
>> . And also perform some operations for backup and check pointing node.
>> A possible solution is to add more main memory but there are certain
>> issues with this approach . Namnenode being Java application, garbage
>> collection cycles execute periodically to reclaim unreferenced heap
>> space. When the heap space grows very large, despite of GC policy
>> chosen, application stalls during the GC activity. This creates a
>> bunch of issues since DNs and  clients may perceive this stall as NN
>> crash.
>> . There will always be a practical limit on how much physical memory a
>> single machine can accommodate.
>> Proposed Solution:
>> Out of the three responsibilities listed above, we can refactor
>> namespace management from the namenode codebase in such a way that
>> there is provision to implement and plug other name systems other than
>> existing in-process memory-based name system. Particularly a name
>> system backed by a distributed key-value store will significantly
>> reduce namenode memory requirement.To achieve this, a new generic
>> interface will be introduced [Let's call it AbstractNameSystem] which
>> defines set of operations using which we perform the namespace
>> management. Namenode code that used to manipulate some java objects
>> maintained in namenode's heap will now operate on this interface.
>> There will be provision for others to extend this interface and plug
> their own NameSystem implementation.
>> To get started, we have implemented the same memory-based namespace
>> implementation in a remote process, outside of the namenode JVM. In
>> addition, work is undergoing to implement the namesystem using HBase.
>> Details of Changes:
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB