Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> [Discuss] Merge federation branch HDFS-1052 into trunk


+
Suresh Srinivas 2011-04-22, 16:48
+
Dhruba Borthakur 2011-04-23, 08:08
+
Doug Cutting 2011-04-25, 21:36
+
suresh srinivas 2011-04-26, 17:29
+
suresh srinivas 2011-04-26, 23:06
+
Doug Cutting 2011-04-27, 04:43
+
Konstantin Shvachko 2011-04-27, 05:26
+
suresh srinivas 2011-04-27, 06:34
+
suresh srinivas 2011-04-27, 06:55
+
suresh srinivas 2011-04-27, 17:02
+
Tsz Wo \ 2011-04-27, 17:08
+
Devaraj Das 2011-04-27, 17:08
+
Konstantin Boudnik 2011-04-27, 17:41
+
suresh srinivas 2011-04-27, 21:36
Copy link to this message
-
Re: [Discuss] Merge federation branch HDFS-1052 into trunk
Suresh,
Showing no degradation in performance on one-node cluster is a good start
for benchmarking.
You still have a dev cluster to run benchmarks, don't you?
--Konstantin

On Wed, Apr 27, 2011 at 2:36 PM, suresh srinivas <[EMAIL PROTECTED]>wrote:

> I ran these tests on my laptop. I would like to use this data to emphasize
> that there is no regression in performance. I am not sure with just the
> tests that I ran we could conclude there is a huge gain in performance with
> federation. When out performance test team runs tests at scale we will get
> more clearer picture.
>
>
>
> On Wed, Apr 27, 2011 at 10:41 AM, Konstantin Boudnik <[EMAIL PROTECTED]
> >wrote:
>
> > Interesting... while the read performance has only marginally improved
> > <4% (still a good thing) the write performance shows significantly
> > better improvements >10%. Very interesting asymmetry, indeed.
> >
> > Suresh, what was the size of the cluster in the testing?
> >  Cos
> >
> > On Wed, Apr 27, 2011 at 10:02, suresh srinivas <[EMAIL PROTECTED]>
> > wrote:
> > > I posted the TestDFSIO comparison with and without federation to
> > HDFS-1052.
> > > Please let me know if it addresses your concern. I am also adding it
> > here:
> > >
> > > TestDFSIO read tests
> > > *Without federation:*
> > > ----- TestDFSIO ----- : read
> > >           Date & time: Wed Apr 27 02:04:24 PDT 2011
> > >       Number of files: 1000
> > > Total MBytes processed: 30000.0
> > >     Throughput mb/sec: 43.62329251162561
> > > Average IO rate mb/sec: 44.619869232177734
> > >  IO rate std deviation: 5.060306158158443
> > >    Test exec time sec: 959.943
> > >
> > > *With federation:*
> > > ----- TestDFSIO ----- : read
> > >           Date & time: Wed Apr 27 02:43:10 PDT 2011
> > >       Number of files: 1000
> > > Total MBytes processed: 30000.0
> > >     Throughput mb/sec: 45.657513857055456
> > > Average IO rate mb/sec: 46.72107696533203
> > >  IO rate std deviation: 5.455125923399539
> > >    Test exec time sec: 924.922
> > >
> > > TestDFSIO write tests
> > > *Without federation:*
> > > ----- TestDFSIO ----- : write
> > >           Date & time: Wed Apr 27 01:47:50 PDT 2011
> > >       Number of files: 1000
> > > Total MBytes processed: 30000.0
> > >     Throughput mb/sec: 35.940755259031015
> > > Average IO rate mb/sec: 38.236236572265625
> > >  IO rate std deviation: 5.929484960036511
> > >    Test exec time sec: 1266.624
> > >
> > > *With federation:*
> > > ----- TestDFSIO ----- : write
> > >           Date & time: Wed Apr 27 02:27:12 PDT 2011
> > >       Number of files: 1000
> > > Total MBytes processed: 30000.0
> > >     Throughput mb/sec: 42.17884674597227
> > > Average IO rate mb/sec: 43.11423873901367
> > >  IO rate std deviation: 5.357057259968647
> > >    Test exec time sec: 1135.298
> > > {noformat}
> > >
> > >
> > > On Tue, Apr 26, 2011 at 11:55 PM, suresh srinivas <
> [EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Konstantin,
> > >>
> > >> Could you provide me link to how this was done on a big feature, like
> > say
> > >> append and how benchmark info was captured? I am planning to run dfsio
> > >> tests, btw.
> > >>
> > >> Regards,
> > >> Suresh
> > >>
> > >>
> > >> On Tue, Apr 26, 2011 at 11:34 PM, suresh srinivas <
> [EMAIL PROTECTED]
> > >wrote:
> > >>
> > >>> Konstantin,
> > >>>
> > >>> On Tue, Apr 26, 2011 at 10:26 PM, Konstantin Shvachko <
> > >>> [EMAIL PROTECTED]> wrote:
> > >>>
> > >>>> Suresh, Sanjay.
> > >>>>
> > >>>> 1. I asked for benchmarks many times over the course of different
> > >>>> discussions on the topic.
> > >>>> I don't see any numbers attached to jira, and I was getting the same
> > >>>> response,
> > >>>> Doug just got from you, guys: which is "why would the performance be
> > >>>> worse".
> > >>>> And this is not an argument for me.
> > >>>>
> > >>>
> > >>> We had done testing earlier and had found that performance had not
> > >>> degraded. We are waiting for out performance team to publish the
> > official
> > >>> numbers to post it to the jira. Unfortunately they are busy
+
Hairong 2011-04-27, 17:46
+
Konstantin Shvachko 2011-04-28, 04:56
+
Konstantin Boudnik 2011-04-28, 13:36
+
suresh srinivas 2011-04-28, 18:02
+
Owen OMalley 2011-04-27, 20:53
+
suresh srinivas 2011-04-27, 21:44
+
Konstantin Shvachko 2011-04-28, 05:12
+
Owen OMalley 2011-04-28, 20:33
+
suresh srinivas 2011-04-28, 22:12
+
Konstantin Shvachko 2011-04-29, 06:30
+
Todd Lipcon 2011-05-02, 21:44
+
suresh srinivas 2011-05-03, 02:17
+
Sanjay Radia 2011-04-27, 00:26
+
Konstantin Boudnik 2011-04-27, 00:59
+
Dhruba Borthakur 2011-04-27, 04:27
+
Tsz Wo \ 2011-04-27, 05:16
+
Konstantin Shvachko 2011-04-27, 05:36
+
Konstantin Boudnik 2011-04-27, 05:40
+
suresh srinivas 2011-04-27, 06:28
+
Sanjay Radia 2011-04-27, 14:03
+
Eli Collins 2011-04-27, 21:36
+
suresh srinivas 2011-04-28, 00:22