Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> measuring iops


Copy link to this message
-
Re: measuring iops
I was curious because when a vendor (big storage company) presented they
were offering a hadoop solution. They posted IOPS and I wasn't sure how
they were determining this number....

On Tue, Oct 23, 2012 at 9:19 AM, Michael Segel <[EMAIL PROTECTED]>wrote:

> You have two issues.
>
> 1) You need to know the throughput in terms of data transfer between disks
> and controller cards on the node.
>
> 2) The actual network throughput of having all of the nodes talking to one
> another as fast as they can. This will let you see your real limitations in
> the ToR Switch's fabric.
>
> Not sure why you really want to do this except to test the disk, disk
> controller, and then networking infrastructure of your ToR and then your
> backplane to connect multiple racks....
>
>
> HTH
>
> -Mike
>
> On Oct 23, 2012, at 7:47 AM, Ravi Prakash <[EMAIL PROTECTED]> wrote:
>
> > Do you mean in a cluster being used by users, or as a benchmark to
> measure the maximum?
> >
> > The JMX page <nn:port>/jmx provides some interesting stats, but I'm not
> sure they have what you want. And I'm unaware of other tools which could.
> >
> >
> >
> >
> >
> > ________________________________
> > From: Rita <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]; Ravi Prakash <[EMAIL PROTECTED]>
> > Sent: Monday, October 22, 2012 6:46 PM
> > Subject: Re: measuring iops
> >
> > Is it possible to know how many reads and writes are occurring thru the
> > entire cluster in a consolidated manner -- this does not include
> > replication factors.
> >
> >
> > On Mon, Oct 22, 2012 at 10:28 AM, Ravi Prakash <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi Rita,
> >>
> >> SliveTest can help you measure the number of reads / writes / deletes /
> ls
> >> / appends per second your NameNode can handle.
> >>
> >> DFSIO can be used to help you measure the amount of throughput.
> >>
> >> Both these tests are actually very flexible and have a plethora of
> options
> >> to help you test different facets of performance. In my experience, you
> >> actually have to be very careful and understand what the tests are doing
> >> for the results to be sensible.
> >>
> >> HTH
> >> Ravi
> >>
> >>
> >>
> >>
> >> ________________________________
> >>   From: Rita <[EMAIL PROTECTED]>
> >> To: "<[EMAIL PROTECTED]>" <[EMAIL PROTECTED]>
> >> Sent: Monday, October 22, 2012 7:23 AM
> >> Subject: Re: measuring iops
> >>
> >> Anyone?
> >>
> >>
> >> On Sun, Oct 21, 2012 at 8:30 AM, Rita <[EMAIL PROTECTED]> wrote:
> >>
> >>> Hi,
> >>>
> >>> Was curious if there was a method to measure the total number of IOPS
> >> (I/O
> >>> operations per second) on a HDFS cluster.
> >>>
> >>>
> >>>
> >>> --
> >>> --- Get your facts first, then you can distort them as you please.--
> >>>
> >>
> >>
> >>
> >> --
> >> --- Get your facts first, then you can distort them as you please.--
> >>
> >
> >
> >
> > --
> > --- Get your facts first, then you can distort them as you please.--
>
>
--
--- Get your facts first, then you can distort them as you please.--
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB