Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> [Discuss] Merge federation branch HDFS-1052 into trunk


Copy link to this message
-
Re: [Discuss] Merge federation branch HDFS-1052 into trunk
Interesting... while the read performance has only marginally improved
<4% (still a good thing) the write performance shows significantly
better improvements >10%. Very interesting asymmetry, indeed.

Suresh, what was the size of the cluster in the testing?
  Cos

On Wed, Apr 27, 2011 at 10:02, suresh srinivas <[EMAIL PROTECTED]> wrote:
> I posted the TestDFSIO comparison with and without federation to HDFS-1052.
> Please let me know if it addresses your concern. I am also adding it here:
>
> TestDFSIO read tests
> *Without federation:*
> ----- TestDFSIO ----- : read
>           Date & time: Wed Apr 27 02:04:24 PDT 2011
>       Number of files: 1000
> Total MBytes processed: 30000.0
>     Throughput mb/sec: 43.62329251162561
> Average IO rate mb/sec: 44.619869232177734
>  IO rate std deviation: 5.060306158158443
>    Test exec time sec: 959.943
>
> *With federation:*
> ----- TestDFSIO ----- : read
>           Date & time: Wed Apr 27 02:43:10 PDT 2011
>       Number of files: 1000
> Total MBytes processed: 30000.0
>     Throughput mb/sec: 45.657513857055456
> Average IO rate mb/sec: 46.72107696533203
>  IO rate std deviation: 5.455125923399539
>    Test exec time sec: 924.922
>
> TestDFSIO write tests
> *Without federation:*
> ----- TestDFSIO ----- : write
>           Date & time: Wed Apr 27 01:47:50 PDT 2011
>       Number of files: 1000
> Total MBytes processed: 30000.0
>     Throughput mb/sec: 35.940755259031015
> Average IO rate mb/sec: 38.236236572265625
>  IO rate std deviation: 5.929484960036511
>    Test exec time sec: 1266.624
>
> *With federation:*
> ----- TestDFSIO ----- : write
>           Date & time: Wed Apr 27 02:27:12 PDT 2011
>       Number of files: 1000
> Total MBytes processed: 30000.0
>     Throughput mb/sec: 42.17884674597227
> Average IO rate mb/sec: 43.11423873901367
>  IO rate std deviation: 5.357057259968647
>    Test exec time sec: 1135.298
> {noformat}
>
>
> On Tue, Apr 26, 2011 at 11:55 PM, suresh srinivas <[EMAIL PROTECTED]>wrote:
>
>> Konstantin,
>>
>> Could you provide me link to how this was done on a big feature, like say
>> append and how benchmark info was captured? I am planning to run dfsio
>> tests, btw.
>>
>> Regards,
>> Suresh
>>
>>
>> On Tue, Apr 26, 2011 at 11:34 PM, suresh srinivas <[EMAIL PROTECTED]>wrote:
>>
>>> Konstantin,
>>>
>>> On Tue, Apr 26, 2011 at 10:26 PM, Konstantin Shvachko <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Suresh, Sanjay.
>>>>
>>>> 1. I asked for benchmarks many times over the course of different
>>>> discussions on the topic.
>>>> I don't see any numbers attached to jira, and I was getting the same
>>>> response,
>>>> Doug just got from you, guys: which is "why would the performance be
>>>> worse".
>>>> And this is not an argument for me.
>>>>
>>>
>>> We had done testing earlier and had found that performance had not
>>> degraded. We are waiting for out performance team to publish the official
>>> numbers to post it to the jira. Unfortunately they are busy qualifying 2xx
>>> releases currently. I will get the perf numbers and post them.
>>>
>>>
>>>>
>>>> 2. I assume that merging requires a vote. I am sure people who know
>>>> bylaws
>>>> better than I do will correct me if it is not true.
>>>> Did I miss the vote?
>>>>
>>>
>>>
>>> As regards to voting, since I was not sure about the procedure, I had
>>> consulted Owen about it. He had indicated that voting is not necessary. If
>>> the right procedure is to call for voting, I will do so. Owen any comments?
>>>
>>>
>>>>
>>>> It feels like you are rushing this and are not doing what you would
>>>> expect
>>>> others to
>>>> do in the same position, and what has been done in the past for such
>>>> large
>>>> projects.
>>>>
>>>
>>> I am not trying to rush here and not follow the procedure required. I am
>>> not sure about what the procedure is. Any pointers to it is appreciated.
>>>
>>>
>>>>
>>>> Thanks,
>>>> --Konstantin
>>>>
>>>>
>>>> On Tue, Apr 26, 2011 at 9:43 PM, Doug Cutting <[EMAIL PROTECTED]>
>>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB