Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> [Proposal] Pluggable Namespace


Copy link to this message
-
Re: [Proposal] Pluggable Namespace
Thanks Bobby.

Experimentation with new namespace implementations and parallel development
is one of the main intents of starting this project from my end.

HDFS has improved a lot, and many of the perceived limitations, such as HA,
Performance, Snapshots, (limited) NFS connectivity have been addressed in
the last two years. I think the namespace scalability is the only checkbox
on that list which has not been fully checked. IMHO, allowing namespaces to
be pluggable, will allow folks to address that.

And I would like to state once again, that this work is orthogonal to
namenode federation, and co-exist with it.

- Milind
---
Milind Bhandarkar
Chief Scientist
Pivotal
+1-650-523-3858 (W)
+1-408-666-8483 (C)
On Mon, Oct 7, 2013 at 8:52 AM, Bobby Evans <[EMAIL PROTECTED]> wrote:

> Putting all conspiracy theories aside :).  Any way we decided to scale the
> name node is going to have limitations.  Federation currently has the
> problem that we cannot easily move data between different name nodes.  It
> is a static partitioning. It is not a blocker, but it can be annoying.  We
> can fix this, but to do so would require some sophisticated coordination
> between the name nodes involved.  If we put the namespace in a key/value
> store like Hbase there are likely to be mapping issues between a tree
> structure and a flat structure making some use cases, like very deep
> trees, potentially a lot slower.  It also does not scale the maximum
> number of operations per second a file system can do.  Because each has
> advantages and drawbacks it is important for us to enabled different use
> cases. This will allow for experimentation and parallel development and
> testing of new namespaces.  I though this was the original vision of
> federation.  Something where /tmp and /archive both co-exist together, but
> potentially have very different implementations to optimize for different
> use cases.
>
>
> Vinod,
>
> Yes block management has been separated out.  This is not about that, it
> is about providing a clean plugin point where someone can more easily take
> advantage of not just the block management code, but also the RPC and
> client code.
>
> --Bobby
>
>
>
> On 10/6/13 10:04 PM, "Mahadev Konar" <[EMAIL PROTECTED]> wrote:
>
> >Milind,
> > Am I missing something here? This was supposed to be a discussion and am
> >hoping thats why you started the thread. I don't see anywhere any
> >conspiracy theory being considered or being talked about. Vinod asked
> >some questions, if you can't or do not want to respond I suggest you skip
> >emailing or ignore rather than making false assumptions and accusations.
> >I hope the intent here is to contribute code and stays that way.
> >
> >thanks
> >mahadev
> >
> >On Oct 6, 2013, at 5:58 PM, Milind Bhandarkar <[EMAIL PROTECTED]>
> >wrote:
> >
> >> Vinod,
> >>
> >> I have received a few emails about concerns that this effort somehow
> >> conflicts with federated namenodes. Most of these emails are from folks
> >> who are directly or remotely associated with Hortonworks.
> >>
> >> Three weeks ago, I sent emails about this effort to a few  Hadoop
> >> committers who are primarily focused on HDFS, whose email address I had.
> >> While 2 out of those three responded to me, the third person associated
> >> with Hortonworks, did not.
> >>
> >> Is Hortonworks concerned that this proposal conflicts with their
> >> development on federated namenode ? I have explicitly stated that it
> >>does
> >> not, and is orthogonal to federation. But I would like to know if there
> >> are some false assumptions being made about the intent of this
> >> development, and would like to quash any conspiracy theories right now,
> >> before they assume a life of their own.
> >>
> >> Thanks,
> >>
> >> Milind
> >>
> >>
> >> -----Original Message-----
> >> From: Vinod Kumar Vavilapalli [mailto:[EMAIL PROTECTED]]
> >> Sent: Sunday, October 06, 2013 12:21 PM
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: [Proposal] Pluggable Namespace
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB