Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> Merging Namenode Federation feature (HDFS-1052) to trunk

Suresh Srinivas 2011-03-03, 22:41
Allen Wittenauer 2011-03-12, 16:43
Konstantin Shvachko 2011-03-14, 17:28
Dhruba Borthakur 2011-03-14, 17:43
Konstantin Shvachko 2011-03-15, 01:12
Travis Crawford 2011-03-15, 05:36
suresh srinivas 2011-03-15, 06:19
Copy link to this message
Re: Merging Namenode Federation feature (HDFS-1052) to trunk
On Mon, Mar 14, 2011 at 11:19 PM, suresh srinivas <[EMAIL PROTECTED]>wrote:

> Thanks for starting off the discussion.
> > This is a huge new feature with 86 jiras already filed, which
> substantially increases the complexity of the code base.
> These are 86 jiras file in a feature branch. We decided to make these
> changes, in smaller increments, instead of a jumbo patch. This was done in
> good faith, as community did not want a jumbo patch (as seen in several
> discussions), to make reviewing of the patch easy and to record the changes
> for reference.

Thanks for doing it that way.
> > Having an in-depth motivation and benchmarking will be needed before the
> community decides on adopting it for support.
> This comes as a surprise, especially from Konstantin :-). The first part of
> the proposal and design both cover motivation.

That is a different motivation. The document talks about why you should use
federation. I am asking about motivation of supporting the code base while
not using it. At least this is how understand Allen's question and some of

So far our tests show no difference with federation.

This is exactly what is needed.
If you could put some numbers in the jira for the reference.

Also it is interesting to know whether there is a benefit in splitting
the namespace. Can I e.g. do more getBlockLocations per second?
This is one of the aspects of scaling, right?
> As we developed this feature, some significant improvements have been made
> to the system - fast snapshots (snapshot time down from 1hr 45 mins to 1
> min!), fast startup, cleanup of storage, fixing multi threading issues in
> several places, decommissioning improvements etc.

This is motivation. I am glad I asked.
> > The purpose of my reply was to get this discussion going, as I found
> Allens question unanswered for 2 weeks.
> My email was sent on March 3rd. Allen's email was sent on March 12th.

Sorry, my bad.
> > The concern he has seems legitimate to me. If ops think federation will
> "make running a grid much much harder" I want to know why and how much
> harder.
> I would like to understand the concerns here. Allen please add details.
> > The way I see it now, Federation introduces
> > - lots of code complexity to the system
> > - harder manageability, according to Allen
> > - potential performance degradation (tbd)
> I have addressed these already.
> > And the main question for those 95% of users, who don't run large
> clusters
> or don't want to place all their compute resources in one data center, is
> what is the advantage in supporting it?
> This is a valid concern. Hence the single namenode configuration that most
> installations run today, will run as is. We put a lot of development and
> testing effort to ensure this.

I don't know what you mean by "as is". My experience with this word in real
estate tells me it can be anything.
suresh srinivas 2011-03-16, 23:54
suresh srinivas 2011-03-16, 23:55
Sanjay Radia 2011-03-14, 17:57
Sanjay Radia 2011-03-21, 23:08
Brian Bockelman 2011-03-21, 23:25
suresh srinivas 2011-03-24, 09:28
Allen Wittenauer 2011-03-22, 20:11
suresh srinivas 2011-03-24, 09:34