Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - [Proposal] Pluggable Namespace


Copy link to this message
-
Re: [Proposal] Pluggable Namespace
Milind Bhandarkar 2013-10-07, 17:18
Thanks Bobby.

Experimentation with new namespace implementations and parallel development
is one of the main intents of starting this project from my end.

HDFS has improved a lot, and many of the perceived limitations, such as HA,
Performance, Snapshots, (limited) NFS connectivity have been addressed in
the last two years. I think the namespace scalability is the only checkbox
on that list which has not been fully checked. IMHO, allowing namespaces to
be pluggable, will allow folks to address that.

And I would like to state once again, that this work is orthogonal to
namenode federation, and co-exist with it.

- Milind
---
Milind Bhandarkar
Chief Scientist
Pivotal
+1-650-523-3858 (W)
+1-408-666-8483 (C)
On Mon, Oct 7, 2013 at 8:52 AM, Bobby Evans <[EMAIL PROTECTED]> wrote:

> Putting all conspiracy theories aside :).  Any way we decided to scale the
> name node is going to have limitations.  Federation currently has the
> problem that we cannot easily move data between different name nodes.  It
> is a static partitioning. It is not a blocker, but it can be annoying.  We
> can fix this, but to do so would require some sophisticated coordination
> between the name nodes involved.  If we put the namespace in a key/value
> store like Hbase there are likely to be mapping issues between a tree
> structure and a flat structure making some use cases, like very deep
> trees, potentially a lot slower.  It also does not scale the maximum
> number of operations per second a file system can do.  Because each has
> advantages and drawbacks it is important for us to enabled different use
> cases. This will allow for experimentation and parallel development and
> testing of new namespaces.  I though this was the original vision of
> federation.  Something where /tmp and /archive both co-exist together, but
> potentially have very different implementations to optimize for different
> use cases.
>
>
> Vinod,
>
> Yes block management has been separated out.  This is not about that, it
> is about providing a clean plugin point where someone can more easily take
> advantage of not just the block management code, but also the RPC and
> client code.
>
> --Bobby
>
>
>
> On 10/6/13 10:04 PM, "Mahadev Konar" <[EMAIL PROTECTED]> wrote:
>
> >Milind,
> > Am I missing something here? This was supposed to be a discussion and am
> >hoping thats why you started the thread. I don't see anywhere any
> >conspiracy theory being considered or being talked about. Vinod asked
> >some questions, if you can't or do not want to respond I suggest you skip
> >emailing or ignore rather than making false assumptions and accusations.
> >I hope the intent here is to contribute code and stays that way.
> >
> >thanks
> >mahadev
> >
> >On Oct 6, 2013, at 5:58 PM, Milind Bhandarkar <[EMAIL PROTECTED]>
> >wrote:
> >
> >> Vinod,
> >>
> >> I have received a few emails about concerns that this effort somehow
> >> conflicts with federated namenodes. Most of these emails are from folks
> >> who are directly or remotely associated with Hortonworks.
> >>
> >> Three weeks ago, I sent emails about this effort to a few  Hadoop
> >> committers who are primarily focused on HDFS, whose email address I had.
> >> While 2 out of those three responded to me, the third person associated
> >> with Hortonworks, did not.
> >>
> >> Is Hortonworks concerned that this proposal conflicts with their
> >> development on federated namenode ? I have explicitly stated that it
> >>does
> >> not, and is orthogonal to federation. But I would like to know if there
> >> are some false assumptions being made about the intent of this
> >> development, and would like to quash any conspiracy theories right now,
> >> before they assume a life of their own.
> >>
> >> Thanks,
> >>
> >> Milind
> >>
> >>
> >> -----Original Message-----
> >> From: Vinod Kumar Vavilapalli [mailto:[EMAIL PROTECTED]]
> >> Sent: Sunday, October 06, 2013 12:21 PM
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: [Proposal] Pluggable Namespace