Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> How to test the performance of NN?


Copy link to this message
-
RE: How to test the performance of NN?
Hi Mark,

Your NNBench output does not look OK to me. Below is an example NNBench cmd line I used some time ago and its output.

Btw, what I found useful  in testing the disk IO performance for namenode operations (others might find this unusual) is the TestEditLog unittest. I would modify the test as appropriate and run the simulation. This was a quick micro-benchmark I used to compare a couple of devices. Note that you'll have to modify the test in trunk and remove the line with EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you're not testing the real thing.
hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20 -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1
-------------- NNBench -------------- :
                                Version: NameNode Benchmark 0.4
                            Date & time: 2012-12-18 07:29:20,832

                         Test Operation: create_write
                             Start time: 2012-12-18 07:25:36,111
                            Maps to run: 32
                         Reduces to run: 1
                     Block Size (bytes): 1
                         Bytes to write: 20
                     Bytes per checksum: 1
                        Number of files: 500
                     Replication factor: 1
             Successful file operations: 16000

         # maps that missed the barrier: 0
                           # exceptions: 0

                TPS: Create/Write/Close: 153
Avg exec time (ms): Create/Write/Close: 301.6148125
             Avg Lat (ms): Create/Write: 68.25625
                    Avg Lat (ms): Close: 233.245

                  RAW DATA: AL Total #1: 1092100
                  RAW DATA: AL Total #2: 3731920
               RAW DATA: TPS Total (ms): 4825837
        RAW DATA: Longest Map Time (ms): 208692.0
                    RAW DATA: Late maps: 0
              RAW DATA: # of exceptions: 0

Hope this helps,
Ivan

From: Mark Kerzner [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 05, 2013 5:16 PM
To: Hadoop User
Subject: Re: How to test the performance of NN?

Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more efficient NN? - I am experimenting with Kove<http://kove.com/> device, persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install just to get experience running it. Then I will run it on a 3-node cluster without Kove, and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should watch for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench:                                Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench:                            Date & time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:                         Test Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench:                             Start time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench:                            Maps to run: 12
13/06/05 16:50:16 INFO hdfs.NNBench:                         Reduces to run: 6
13/06/05 16:50:16 INFO hdfs.NNBench:                     Block Size (bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench:                         Bytes to write: 0
13/06/05 16:50:16 INFO hdfs.NNBench:                     Bytes per checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench:                        Number of files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench:                     Replication factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench:             Successful file operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:         # maps that missed the barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench:                           # exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:                         TPS: Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench:          Avg Exec time (ms): Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench:                     Avg Lat (ms): Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench:                   Avg Lat (ms): Read: NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:                  RAW DATA: AL Total #1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench:                  RAW DATA: AL Total #2: 0
13/06/05 16:50:16 INFO hdfs.NNBench:               RAW DATA: TPS Total (ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench:        RAW DATA: Longest Map Time (ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench:                    RAW DATA: Late maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench:              RAW DATA: # of exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Mark,

NNBench is a namenode load test. Output of the test is the set of performance numbers, like transactions per second, average latency of operations, etc.

What do you mean by trying to create a more efficient namenode? What dimension are you trying to optimize? Depending on this, people on this list might be able to guide you to something more appropriate.

Hope this helps,
Ivan

From: Mark Kerzner [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?

Hi,

I am trying to create a more efficient namenode, and for that I need to the standard distribution, and then compare it to my version.

Which benchmark should I run? I am doing nnbench, but it is not telling m