Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> Merging Namenode Federation feature (HDFS-1052) to trunk


+
Suresh Srinivas 2011-03-03, 22:41
+
Allen Wittenauer 2011-03-12, 16:43
+
Konstantin Shvachko 2011-03-14, 17:28
+
Dhruba Borthakur 2011-03-14, 17:43
Copy link to this message
-
Re: Merging Namenode Federation feature (HDFS-1052) to trunk
Dhruba, good you are speaking up for federation.
I consider it important as it means more support for the feature in the
future.

The purpose of my reply was to get this discussion going, as I found Allens
question unanswered for 2 weeks.
The concern he has seems legitimate to me. If ops think federation will
"make running a grid much much harder" I want to know why and how much
harder.
Because cluster "manageability" is claimed as one of the objectives of
federation.

I sure am well familiar with the design being a part of it for a while.
And all my concerns have been articulated and well known. Though not all of
them are addressed.

The way I see it now, Federation introduces
- lots of code complexity to the system
- harder manageability, according to Allen
- potential performance degradation (tbd)
And the main question for those 95% of users, who don't run large clusters
or
don't want to place all their compute resources in one data center, is what
is the advantage in supporting it?

Performance-wise there 2 main aspects:
- Does federation give me the same cluster performance if I don't federate?
- If I federate how much more throughput can I get?

Thanks,
--Konstantin

On Mon, Mar 14, 2011 at 10:43 AM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:

> Hi folks,
>
> The design for the federation work has been a published and there is a very
> well-written design document. It explains the pros-and-cons of each design
> point. It would be nice if more people can review this document and provide
> comments on how to make it better. The implementation is in progress but
> that does not mean that the
> "design-is-cast-in-stone-and-cannot-be-enhanced".
>
> Allen: can you pl describe what you mean by  "It sounds like merging into
> trunk is extremely premature". If we can make all unit tests pass
> successfully on the branch, then do you think we should merge that branch
> into the trunk?
>
> Konstantin: I agree that federation introduces new code complexity. But it
> is a fact that introducing a new heavy-weight feature will add complexity.
> If you have a different proposal (and implementation) to scale namenode,
> please share it with us and we can then evaluate these designs in terms on
> complexity/feature. If you have questions about certain issues in the
> design, it would be great if you can ask them now. Hopefully, the folks
> doing the implementation can then provide you performance numbers to
> alleviate your concerns.
>
> From that way I look at it, I think the federation-feature is a huge
> positive step in the right direction.
>
> thanks,
> dhruba
>
>
>
>
> On Mon, Mar 14, 2011 at 10:28 AM, Konstantin Shvachko <
> [EMAIL PROTECTED]> wrote:
>
>> Allen is right.
>> This is a huge new feature with 86 jiras already filed, which
>> substantially
>> increases the complexity of the code base.
>> Having an in-depth motivation and benchmarking will be needed before the
>> community decides on adopting it for support.
>> Thanks,
>> --Konstantin
>>
>>
>>
>> On Sat, Mar 12, 2011 at 8:43 AM, Allen Wittenauer
>> <[EMAIL PROTECTED]>wrote:
>>
>> >
>> > On Mar 3, 2011, at 2:41 PM, Suresh Srinivas wrote:
>> >
>> > > We have started pushing changes for namenode federation in to the
>> feature
>> > branch HDFS-1052. The work items are created as subtask of the jira
>> > HDFS-1052 and are based on the design document published in the same
>> jira.
>> > By the end of this week, we will complete pushing the changes to
>> HDFS-1052
>> > branch. Though the changes in these jiras are already committed, please
>> do
>> > provide your feedback on either HDFS-1052 or its subtasks. New items
>> that
>> > come out of the feedback will be addressed in new jiras.
>> >
>> > >
>> > > Current status of the development:
>> > > # The testing of this feature is underway. Most of the basic
>> > functionality has been tested both for a single namenode cluster (for
>> > backward compatibility) and with multiple namenodes.
>> > > # All the existing tests and newly added tests pass (same as trunk).
+
Travis Crawford 2011-03-15, 05:36
+
suresh srinivas 2011-03-15, 06:19
+
Konstantin Shvachko 2011-03-16, 22:52
+
suresh srinivas 2011-03-16, 23:54
+
suresh srinivas 2011-03-16, 23:55
+
Sanjay Radia 2011-03-14, 17:57
+
Sanjay Radia 2011-03-21, 23:08
+
Brian Bockelman 2011-03-21, 23:25
+
suresh srinivas 2011-03-24, 09:28
+
Allen Wittenauer 2011-03-22, 20:11
+
suresh srinivas 2011-03-24, 09:34
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB