Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - [Discuss] Merge federation branch HDFS-1052 into trunk


Copy link to this message
-
Re: [Discuss] Merge federation branch HDFS-1052 into trunk
Konstantin Shvachko 2011-04-28, 05:18
Suresh,
Showing no degradation in performance on one-node cluster is a good start
for benchmarking.
You still have a dev cluster to run benchmarks, don't you?
--Konstantin

On Wed, Apr 27, 2011 at 2:36 PM, suresh srinivas <[EMAIL PROTECTED]>wrote:

> I ran these tests on my laptop. I would like to use this data to emphasize
> that there is no regression in performance. I am not sure with just the
> tests that I ran we could conclude there is a huge gain in performance with
> federation. When out performance test team runs tests at scale we will get
> more clearer picture.
>
>
>
> On Wed, Apr 27, 2011 at 10:41 AM, Konstantin Boudnik <[EMAIL PROTECTED]
> >wrote:
>
> > Interesting... while the read performance has only marginally improved
> > <4% (still a good thing) the write performance shows significantly
> > better improvements >10%. Very interesting asymmetry, indeed.
> >
> > Suresh, what was the size of the cluster in the testing?
> >  Cos
> >
> > On Wed, Apr 27, 2011 at 10:02, suresh srinivas <[EMAIL PROTECTED]>
> > wrote:
> > > I posted the TestDFSIO comparison with and without federation to
> > HDFS-1052.
> > > Please let me know if it addresses your concern. I am also adding it
> > here:
> > >
> > > TestDFSIO read tests
> > > *Without federation:*
> > > ----- TestDFSIO ----- : read
> > >           Date & time: Wed Apr 27 02:04:24 PDT 2011
> > >       Number of files: 1000
> > > Total MBytes processed: 30000.0
> > >     Throughput mb/sec: 43.62329251162561
> > > Average IO rate mb/sec: 44.619869232177734
> > >  IO rate std deviation: 5.060306158158443
> > >    Test exec time sec: 959.943
> > >
> > > *With federation:*
> > > ----- TestDFSIO ----- : read
> > >           Date & time: Wed Apr 27 02:43:10 PDT 2011
> > >       Number of files: 1000
> > > Total MBytes processed: 30000.0
> > >     Throughput mb/sec: 45.657513857055456
> > > Average IO rate mb/sec: 46.72107696533203
> > >  IO rate std deviation: 5.455125923399539
> > >    Test exec time sec: 924.922
> > >
> > > TestDFSIO write tests
> > > *Without federation:*
> > > ----- TestDFSIO ----- : write
> > >           Date & time: Wed Apr 27 01:47:50 PDT 2011
> > >       Number of files: 1000
> > > Total MBytes processed: 30000.0
> > >     Throughput mb/sec: 35.940755259031015
> > > Average IO rate mb/sec: 38.236236572265625
> > >  IO rate std deviation: 5.929484960036511
> > >    Test exec time sec: 1266.624
> > >
> > > *With federation:*
> > > ----- TestDFSIO ----- : write
> > >           Date & time: Wed Apr 27 02:27:12 PDT 2011
> > >       Number of files: 1000
> > > Total MBytes processed: 30000.0
> > >     Throughput mb/sec: 42.17884674597227
> > > Average IO rate mb/sec: 43.11423873901367
> > >  IO rate std deviation: 5.357057259968647
> > >    Test exec time sec: 1135.298
> > > {noformat}
> > >
> > >
> > > On Tue, Apr 26, 2011 at 11:55 PM, suresh srinivas <
> [EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Konstantin,
> > >>
> > >> Could you provide me link to how this was done on a big feature, like
> > say
> > >> append and how benchmark info was captured? I am planning to run dfsio
> > >> tests, btw.
> > >>
> > >> Regards,
> > >> Suresh
> > >>
> > >>
> > >> On Tue, Apr 26, 2011 at 11:34 PM, suresh srinivas <
> [EMAIL PROTECTED]
> > >wrote:
> > >>
> > >>> Konstantin,
> > >>>
> > >>> On Tue, Apr 26, 2011 at 10:26 PM, Konstantin Shvachko <
> > >>> [EMAIL PROTECTED]> wrote:
> > >>>
> > >>>> Suresh, Sanjay.
> > >>>>
> > >>>> 1. I asked for benchmarks many times over the course of different
> > >>>> discussions on the topic.
> > >>>> I don't see any numbers attached to jira, and I was getting the same
> > >>>> response,
> > >>>> Doug just got from you, guys: which is "why would the performance be
> > >>>> worse".
> > >>>> And this is not an argument for me.
> > >>>>
> > >>>
> > >>> We had done testing earlier and had found that performance had not
> > >>> degraded. We are waiting for out performance team to publish the
> > official
> > >>> numbers to post it to the jira. Unfortunately they are busy