Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> Merging Namenode Federation feature (HDFS-1052) to trunk


+
Suresh Srinivas 2011-03-03, 22:41
+
Allen Wittenauer 2011-03-12, 16:43
+
Konstantin Shvachko 2011-03-14, 17:28
+
Dhruba Borthakur 2011-03-14, 17:43
Copy link to this message
-
Re: Merging Namenode Federation feature (HDFS-1052) to trunk
Dhruba, good you are speaking up for federation.
I consider it important as it means more support for the feature in the
future.

The purpose of my reply was to get this discussion going, as I found Allens
question unanswered for 2 weeks.
The concern he has seems legitimate to me. If ops think federation will
"make running a grid much much harder" I want to know why and how much
harder.
Because cluster "manageability" is claimed as one of the objectives of
federation.

I sure am well familiar with the design being a part of it for a while.
And all my concerns have been articulated and well known. Though not all of
them are addressed.

The way I see it now, Federation introduces
- lots of code complexity to the system
- harder manageability, according to Allen
- potential performance degradation (tbd)
And the main question for those 95% of users, who don't run large clusters
or
don't want to place all their compute resources in one data center, is what
is the advantage in supporting it?

Performance-wise there 2 main aspects:
- Does federation give me the same cluster performance if I don't federate?
- If I federate how much more throughput can I get?

Thanks,
--Konstantin

On Mon, Mar 14, 2011 at 10:43 AM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:

> Hi folks,
>
> The design for the federation work has been a published and there is a very
> well-written design document. It explains the pros-and-cons of each design
> point. It would be nice if more people can review this document and provide
> comments on how to make it better. The implementation is in progress but
> that does not mean that the
> "design-is-cast-in-stone-and-cannot-be-enhanced".
>
> Allen: can you pl describe what you mean by  "It sounds like merging into
> trunk is extremely premature". If we can make all unit tests pass
> successfully on the branch, then do you think we should merge that branch
> into the trunk?
>
> Konstantin: I agree that federation introduces new code complexity. But it
> is a fact that introducing a new heavy-weight feature will add complexity.
> If you have a different proposal (and implementation) to scale namenode,
> please share it with us and we can then evaluate these designs in terms on
> complexity/feature. If you have questions about certain issues in the
> design, it would be great if you can ask them now. Hopefully, the folks
> doing the implementation can then provide you performance numbers to
> alleviate your concerns.
>
> From that way I look at it, I think the federation-feature is a huge
> positive step in the right direction.
>
> thanks,
> dhruba
>
>
>
>
> On Mon, Mar 14, 2011 at 10:28 AM, Konstantin Shvachko <
> [EMAIL PROTECTED]> wrote:
>
>> Allen is right.
>> This is a huge new feature with 86 jiras already filed, which
>> substantially
>> increases the complexity of the code base.
>> Having an in-depth motivation and benchmarking will be needed before the
>> community decides on adopting it for support.
>> Thanks,
>> --Konstantin
>>
>>
>>
>> On Sat, Mar 12, 2011 at 8:43 AM, Allen Wittenauer
>> <[EMAIL PROTECTED]>wrote:
>>
>> >
>> > On Mar 3, 2011, at 2:41 PM, Suresh Srinivas wrote:
>> >
>> > > We have started pushing changes for namenode federation in to the
>> feature
>> > branch HDFS-1052. The work items are created as subtask of the jira
>> > HDFS-1052 and are based on the design document published in the same
>> jira.
>> > By the end of this week, we will complete pushing the changes to
>> HDFS-1052
>> > branch. Though the changes in these jiras are already committed, please
>> do
>> > provide your feedback on either HDFS-1052 or its subtasks. New items
>> that
>> > come out of the feedback will be addressed in new jiras.
>> >
>> > >
>> > > Current status of the development:
>> > > # The testing of this feature is underway. Most of the basic
>> > functionality has been tested both for a single namenode cluster (for
>> > backward compatibility) and with multiple namenodes.
>> > > # All the existing tests and newly added tests pass (same as trunk).
+
Travis Crawford 2011-03-15, 05:36
+
suresh srinivas 2011-03-15, 06:19
+
Konstantin Shvachko 2011-03-16, 22:52
+
suresh srinivas 2011-03-16, 23:54
+
suresh srinivas 2011-03-16, 23:55
+
Sanjay Radia 2011-03-14, 17:57
+
Sanjay Radia 2011-03-21, 23:08
+
Brian Bockelman 2011-03-21, 23:25
+
suresh srinivas 2011-03-24, 09:28
+
Allen Wittenauer 2011-03-22, 20:11
+
suresh srinivas 2011-03-24, 09:34