Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - [Proposal] Pluggable Namespace


Copy link to this message
-
Re: [Proposal] Pluggable Namespace
Konstantin Shvachko 2013-10-08, 02:40
Milind,

Seems as a proper time to open a Jira.
Looks to me nobody is objecting to the general idea of AbstractNamesystem.
As usually the details is what finally matters.

FSNamesystem does have a formal interface called Namesystem now, but
it is somewhat arbitrary and probably rudimentary for your purposes.
We should understand what is abstract and what is current implementation
specific.
Things like block or inode ids, generation stamps, along with namespace
maintenance
separated from block management, which currently interleave with each other,
etc., should be considered.

IMO it is better to keep such technical discussion in a jira for future
reference.

Your implementation of the namespace, LevelDB effort, Giraffa, our work on
ConsensusNode considered as use cases for AbstractNamesystem can potentially
shape out common requirements for your project.
Thanks,
--Konst
On Mon, Oct 7, 2013 at 4:50 PM, Milind Bhandarkar <[EMAIL PROTECTED]
> wrote:

> Getting back to the technical discussion: based on this proposal email
> that I sent, I came to know, from Sanjay Radia, of a summer intern project
> at Hortonworks that implemented namespace using LevelDB. We discussed it on
> personal email today, and agreed that these two efforts were still
> orthogonal, but will have to be merged because of all the changes that had
> to be made to NN code. Sanjay has promised to post more details on this
> thread. Both the levelDB jira and our jira will be filed soon, and we will
> continue discussions there.
>
>
> Now, if I may respond to Doug's email: The LevelDB impl was apparently
> presented at August Yahoo Hadoop meetup, which I did not attend, and since
> I was away from this country, did not even bother to check which talks were
> scheduled. But, I was surprised that no one responded to my proposal
> pointing to that effort. If I had not known about it until much later into
> implementation cycle, it would have been hard for me to account for the
> merge efforts needed.
>
> How do we keep each other informed what we are planning to do in apache,
> without everyone having to attend non-apache meetups? I think this mailing
> list for proposals is a good choice, but is everyone willing to go on
> record here declaring their plans in advance? In my case, the amount of
> time it took to see whether it was doable, with all the public NS
> modification proposals, to go public with this proposal, was more than
> actually implementing the changes.
>
> I would seek your inputs to see how we can streamline this process,
> especially for major changes, but perhaps on a separate thread.
>
> Thanks,
>
> Milind
>
>
> Sent from my iPhone
>
> >> On Oct 7, 2013, at 12:50, Doug Cutting <[EMAIL PROTECTED]> wrote:
> >>
> >> On Mon, Oct 7, 2013 at 12:05 PM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
> >> I recognize some of what we recently experienced on a HDFS matter in
> what
> >> Milind wrote even if this was not the appropriate forum for it. Odd
> mention
> >> of "conspiracy theories" aside, for people who may come to this thread
> >> later, perhaps you can recommend an appropriate public Apache community
> >> forum for discussing such concerns?
> >
> > I'm not exactly sure what "such concerns" are, but I'll guess that
> > you're worried that someone else has plans that conflict with yours
> > yet they're not stating that publicly.  Is that right?
> >
> > We cannot ensure that everyone's contributions are equally loved by
> > everyone else.  Some people's contributions may sometimes conflict
> > with plans of others.  (I have no idea whether that's the case here.)
> > Making a proposal is a good way to inquire about this.  Directly
> > accusing one group of implied opposition is probably
> > counter-productive.
> >
> > We try to ensure that every contribution is given a fair discussion in
> > public and that it is weighed on its technical merit alone.  If a
> > proposal conflicts with some other not-yet-public plan then it's up to
> > the not-public planners to voluntarily make themselves public and try