Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - [Discuss] Merge federation branch HDFS-1052 into trunk


Copy link to this message
-
Re: [Discuss] Merge federation branch HDFS-1052 into trunk
Konstantin Boudnik 2011-04-28, 13:36
+1. Having an open QE process would be a tremendous value-add to the
overall quality of the feature. Append was an exemplary development in
this sense. Would it be possible to have Federation test plan (if
exists) to be published along with the specs on the JIRA (similar to
HDFS-265) at least for the reference?

Cos

On Wed, Apr 27, 2011 at 21:56, Konstantin Shvachko <[EMAIL PROTECTED]> wrote:
> Yes, I can talk about append as an example.
> Some differences with federation project are:
> - append had a comprehensive test plan document, which was designed an
> executed;
> - append was independently evaluated by HBase guys;
> - it introduced new benchmark for append;
> - We ran both DFSIO and NNThroughput. DFSIO was executed on a relatively
> small cluster. I couldn't find where I posted the results, my bad. But you
> may be able to find these tasks in our scrum records.
>
> --Konstantin
>
>
> On Tue, Apr 26, 2011 at 11:55 PM, suresh srinivas <[EMAIL PROTECTED]>wrote:
>
>> Konstantin,
>>
>> Could you provide me link to how this was done on a big feature, like say
>> append and how benchmark info was captured? I am planning to run dfsio
>> tests, btw.
>>
>> Regards,
>> Suresh
>>
>> On Tue, Apr 26, 2011 at 11:34 PM, suresh srinivas <[EMAIL PROTECTED]
>> >wrote:
>>
>> > Konstantin,
>> >
>> > On Tue, Apr 26, 2011 at 10:26 PM, Konstantin Shvachko <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> Suresh, Sanjay.
>> >>
>> >> 1. I asked for benchmarks many times over the course of different
>> >> discussions on the topic.
>> >> I don't see any numbers attached to jira, and I was getting the same
>> >> response,
>> >> Doug just got from you, guys: which is "why would the performance be
>> >> worse".
>> >> And this is not an argument for me.
>> >>
>> >
>> > We had done testing earlier and had found that performance had not
>> > degraded. We are waiting for out performance team to publish the official
>> > numbers to post it to the jira. Unfortunately they are busy qualifying
>> 2xx
>> > releases currently. I will get the perf numbers and post them.
>> >
>> >
>> >>
>> >> 2. I assume that merging requires a vote. I am sure people who know
>> bylaws
>> >> better than I do will correct me if it is not true.
>> >> Did I miss the vote?
>> >>
>> >
>> >
>> > As regards to voting, since I was not sure about the procedure, I had
>> > consulted Owen about it. He had indicated that voting is not necessary.
>> If
>> > the right procedure is to call for voting, I will do so. Owen any
>> comments?
>> >
>> >
>> >>
>> >> It feels like you are rushing this and are not doing what you would
>> expect
>> >> others to
>> >> do in the same position, and what has been done in the past for such
>> large
>> >> projects.
>> >>
>> >
>> > I am not trying to rush here and not follow the procedure required. I am
>> > not sure about what the procedure is. Any pointers to it is appreciated.
>> >
>> >
>> >>
>> >> Thanks,
>> >> --Konstantin
>> >>
>> >>
>> >> On Tue, Apr 26, 2011 at 9:43 PM, Doug Cutting <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> > Suresh, Sanjay,
>> >> >
>> >> > Thank you very much for addressing my questions.
>> >> >
>> >> > Cheers,
>> >> >
>> >> > Doug
>> >> >
>> >> > On 04/26/2011 10:29 AM, suresh srinivas wrote:
>> >> > > Doug,
>> >> > >
>> >> > >
>> >> > >> 1. Can you please describe the significant advantages this approach
>> >> has
>> >> > >> over a symlink-based approach?
>> >> > >
>> >> > > Federation is complementary with symlink approach. You could choose
>> to
>> >> > > provide integrated namespace using symlinks. However, client side
>> >> mount
>> >> > > tables seems a better approach for many reasons:
>> >> > > # Unlike symbolic links, client side mount tables can choose to go
>> to
>> >> > right
>> >> > > namenode based on configuration. This avoids unnecessary RPCs to the
>> >> > > namenodes to discover the targer of symlink.
>> >> > > # The unavailability of a namenode where a symbolic link is
>> configured
>> >> > does
>> >> > > not affect reaching the symlink target.