Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - measuring iops


+
Rita 2012-10-21, 12:30
+
Rita 2012-10-22, 12:23
+
Ravi Prakash 2012-10-22, 14:28
+
Rita 2012-10-22, 23:46
+
Ravi Prakash 2012-10-23, 12:47
Copy link to this message
-
Re: measuring iops
Michael Segel 2012-10-23, 13:19
You have two issues.

1) You need to know the throughput in terms of data transfer between disks and controller cards on the node.

2) The actual network throughput of having all of the nodes talking to one another as fast as they can. This will let you see your real limitations in the ToR Switch's fabric.

Not sure why you really want to do this except to test the disk, disk controller, and then networking infrastructure of your ToR and then your backplane to connect multiple racks....
HTH

-Mike

On Oct 23, 2012, at 7:47 AM, Ravi Prakash <[EMAIL PROTECTED]> wrote:

> Do you mean in a cluster being used by users, or as a benchmark to measure the maximum?
>
> The JMX page <nn:port>/jmx provides some interesting stats, but I'm not sure they have what you want. And I'm unaware of other tools which could.
>
>
>
>
>
> ________________________________
> From: Rita <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; Ravi Prakash <[EMAIL PROTECTED]>
> Sent: Monday, October 22, 2012 6:46 PM
> Subject: Re: measuring iops
>
> Is it possible to know how many reads and writes are occurring thru the
> entire cluster in a consolidated manner -- this does not include
> replication factors.
>
>
> On Mon, Oct 22, 2012 at 10:28 AM, Ravi Prakash <[EMAIL PROTECTED]> wrote:
>
>> Hi Rita,
>>
>> SliveTest can help you measure the number of reads / writes / deletes / ls
>> / appends per second your NameNode can handle.
>>
>> DFSIO can be used to help you measure the amount of throughput.
>>
>> Both these tests are actually very flexible and have a plethora of options
>> to help you test different facets of performance. In my experience, you
>> actually have to be very careful and understand what the tests are doing
>> for the results to be sensible.
>>
>> HTH
>> Ravi
>>
>>
>>
>>
>> ________________________________
>>   From: Rita <[EMAIL PROTECTED]>
>> To: "<[EMAIL PROTECTED]>" <[EMAIL PROTECTED]>
>> Sent: Monday, October 22, 2012 7:23 AM
>> Subject: Re: measuring iops
>>
>> Anyone?
>>
>>
>> On Sun, Oct 21, 2012 at 8:30 AM, Rita <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> Was curious if there was a method to measure the total number of IOPS
>> (I/O
>>> operations per second) on a HDFS cluster.
>>>
>>>
>>>
>>> --
>>> --- Get your facts first, then you can distort them as you please.--
>>>
>>
>>
>>
>> --
>> --- Get your facts first, then you can distort them as you please.--
>>
>
>
>
> --
> --- Get your facts first, then you can distort them as you please.--
+
Rita 2012-10-23, 23:01
+
Brian Bockelman 2012-10-23, 23:40
+
Lance Norskog 2012-10-26, 11:56