Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> [Discuss] Merge federation branch HDFS-1052 into trunk


Copy link to this message
-
Re: [Discuss] Merge federation branch HDFS-1052 into trunk
Interesting... while the read performance has only marginally improved
<4% (still a good thing) the write performance shows significantly
better improvements >10%. Very interesting asymmetry, indeed.

Suresh, what was the size of the cluster in the testing?
  Cos

On Wed, Apr 27, 2011 at 10:02, suresh srinivas <[EMAIL PROTECTED]> wrote:
> I posted the TestDFSIO comparison with and without federation to HDFS-1052.
> Please let me know if it addresses your concern. I am also adding it here:
>
> TestDFSIO read tests
> *Without federation:*
> ----- TestDFSIO ----- : read
>           Date & time: Wed Apr 27 02:04:24 PDT 2011
>       Number of files: 1000
> Total MBytes processed: 30000.0
>     Throughput mb/sec: 43.62329251162561
> Average IO rate mb/sec: 44.619869232177734
>  IO rate std deviation: 5.060306158158443
>    Test exec time sec: 959.943
>
> *With federation:*
> ----- TestDFSIO ----- : read
>           Date & time: Wed Apr 27 02:43:10 PDT 2011
>       Number of files: 1000
> Total MBytes processed: 30000.0
>     Throughput mb/sec: 45.657513857055456
> Average IO rate mb/sec: 46.72107696533203
>  IO rate std deviation: 5.455125923399539
>    Test exec time sec: 924.922
>
> TestDFSIO write tests
> *Without federation:*
> ----- TestDFSIO ----- : write
>           Date & time: Wed Apr 27 01:47:50 PDT 2011
>       Number of files: 1000
> Total MBytes processed: 30000.0
>     Throughput mb/sec: 35.940755259031015
> Average IO rate mb/sec: 38.236236572265625
>  IO rate std deviation: 5.929484960036511
>    Test exec time sec: 1266.624
>
> *With federation:*
> ----- TestDFSIO ----- : write
>           Date & time: Wed Apr 27 02:27:12 PDT 2011
>       Number of files: 1000
> Total MBytes processed: 30000.0
>     Throughput mb/sec: 42.17884674597227
> Average IO rate mb/sec: 43.11423873901367
>  IO rate std deviation: 5.357057259968647
>    Test exec time sec: 1135.298
> {noformat}
>
>
> On Tue, Apr 26, 2011 at 11:55 PM, suresh srinivas <[EMAIL PROTECTED]>wrote:
>
>> Konstantin,
>>
>> Could you provide me link to how this was done on a big feature, like say
>> append and how benchmark info was captured? I am planning to run dfsio
>> tests, btw.
>>
>> Regards,
>> Suresh
>>
>>
>> On Tue, Apr 26, 2011 at 11:34 PM, suresh srinivas <[EMAIL PROTECTED]>wrote:
>>
>>> Konstantin,
>>>
>>> On Tue, Apr 26, 2011 at 10:26 PM, Konstantin Shvachko <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Suresh, Sanjay.
>>>>
>>>> 1. I asked for benchmarks many times over the course of different
>>>> discussions on the topic.
>>>> I don't see any numbers attached to jira, and I was getting the same
>>>> response,
>>>> Doug just got from you, guys: which is "why would the performance be
>>>> worse".
>>>> And this is not an argument for me.
>>>>
>>>
>>> We had done testing earlier and had found that performance had not
>>> degraded. We are waiting for out performance team to publish the official
>>> numbers to post it to the jira. Unfortunately they are busy qualifying 2xx
>>> releases currently. I will get the perf numbers and post them.
>>>
>>>
>>>>
>>>> 2. I assume that merging requires a vote. I am sure people who know
>>>> bylaws
>>>> better than I do will correct me if it is not true.
>>>> Did I miss the vote?
>>>>
>>>
>>>
>>> As regards to voting, since I was not sure about the procedure, I had
>>> consulted Owen about it. He had indicated that voting is not necessary. If
>>> the right procedure is to call for voting, I will do so. Owen any comments?
>>>
>>>
>>>>
>>>> It feels like you are rushing this and are not doing what you would
>>>> expect
>>>> others to
>>>> do in the same position, and what has been done in the past for such
>>>> large
>>>> projects.
>>>>
>>>
>>> I am not trying to rush here and not follow the procedure required. I am
>>> not sure about what the procedure is. Any pointers to it is appreciated.
>>>
>>>
>>>>
>>>> Thanks,
>>>> --Konstantin
>>>>
>>>>
>>>> On Tue, Apr 26, 2011 at 9:43 PM, Doug Cutting <[EMAIL PROTECTED]>
>>>