Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> [Proposal] Pluggable Namespace

Copy link to this message
Re: [Proposal] Pluggable Namespace
Putting all conspiracy theories aside :).  Any way we decided to scale the
name node is going to have limitations.  Federation currently has the
problem that we cannot easily move data between different name nodes.  It
is a static partitioning. It is not a blocker, but it can be annoying.  We
can fix this, but to do so would require some sophisticated coordination
between the name nodes involved.  If we put the namespace in a key/value
store like Hbase there are likely to be mapping issues between a tree
structure and a flat structure making some use cases, like very deep
trees, potentially a lot slower.  It also does not scale the maximum
number of operations per second a file system can do.  Because each has
advantages and drawbacks it is important for us to enabled different use
cases. This will allow for experimentation and parallel development and
testing of new namespaces.  I though this was the original vision of
federation.  Something where /tmp and /archive both co-exist together, but
potentially have very different implementations to optimize for different
use cases.

Yes block management has been separated out.  This is not about that, it
is about providing a clean plugin point where someone can more easily take
advantage of not just the block management code, but also the RPC and
client code.



On 10/6/13 10:04 PM, "Mahadev Konar" <[EMAIL PROTECTED]> wrote:

> Am I missing something here? This was supposed to be a discussion and am
>hoping thats why you started the thread. I don't see anywhere any
>conspiracy theory being considered or being talked about. Vinod asked
>some questions, if you can't or do not want to respond I suggest you skip
>emailing or ignore rather than making false assumptions and accusations.
>I hope the intent here is to contribute code and stays that way.
>On Oct 6, 2013, at 5:58 PM, Milind Bhandarkar <[EMAIL PROTECTED]>
>> Vinod,
>> I have received a few emails about concerns that this effort somehow
>> conflicts with federated namenodes. Most of these emails are from folks
>> who are directly or remotely associated with Hortonworks.
>> Three weeks ago, I sent emails about this effort to a few  Hadoop
>> committers who are primarily focused on HDFS, whose email address I had.
>> While 2 out of those three responded to me, the third person associated
>> with Hortonworks, did not.
>> Is Hortonworks concerned that this proposal conflicts with their
>> development on federated namenode ? I have explicitly stated that it
>> not, and is orthogonal to federation. But I would like to know if there
>> are some false assumptions being made about the intent of this
>> development, and would like to quash any conspiracy theories right now,
>> before they assume a life of their own.
>> Thanks,
>> Milind
>> -----Original Message-----
>> From: Vinod Kumar Vavilapalli [mailto:[EMAIL PROTECTED]]
>> Sent: Sunday, October 06, 2013 12:21 PM
>> Subject: Re: [Proposal] Pluggable Namespace
>> In order to make federation happen, the block pool management was
>> separated. Isn't that the same as this effortt?
>> Thanks,
>> +Vinod
>> On Oct 6, 2013, at 9:35 AM, Milind Bhandarkar wrote:
>>> Federation is orthogonal with Pluggable Namespaces. That is, one can
>>> use Federation if needed, even while a distributed K-V store is used
>>> on the backend.
>>> Limitations of Federated namenode for scaling namespace are
>>> well-documented in several places, including the Giraffa presentation.
>>> HBase is only one of the several namespace implementations possible.
>>> Thus, if HBase-based namespace implementation does not fit your
>>> performance needs, you have a choice of using something else.
>>> - milind
>>> -----Original Message-----
>>> From: Azuryy Yu [mailto:[EMAIL PROTECTED]]
>>> Sent: Saturday, October 05, 2013 6:41 PM