Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> How to test the performance of NN?


+
Mark Kerzner 2013-06-05, 22:17
+
Ivan Mitic 2013-06-05, 22:32
+
Mark Kerzner 2013-06-06, 00:16
+
Ivan Mitic 2013-06-06, 06:18
Copy link to this message
-
Re: How to test the performance of NN?
Awesome advice, Ivan! It works for me now.

Thank you,
Mark
On Thu, Jun 6, 2013 at 12:18 AM, Ivan Mitic <[EMAIL PROTECTED]> wrote:

>  Hi Mark,****
>
> ** **
>
> Your NNBench output does not look OK to me. Below is an example NNBench
> cmd line I used some time ago and its output.****
>
> ** **
>
> Btw, what I found useful  in testing the disk IO performance for namenode
> operations (others might find this unusual) is the TestEditLog unittest. I
> would modify the test as appropriate and run the simulation. This was a
> quick micro-benchmark I used to compare a couple of devices. Note that
> you’ll have to modify the test in trunk and remove the line with
> EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you’re
> not testing the real thing.****
>
> ** **
>
> ** **
>
> hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation
> create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20
> -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1****
>
> ** **
>
> ** **
>
> -------------- NNBench -------------- :****
>
>                                 Version: NameNode Benchmark 0.4****
>
>                             Date & time: 2012-12-18 07:29:20,832****
>
> ** **
>
>                          Test Operation: create_write****
>
>                              Start time: 2012-12-18 07:25:36,111****
>
>                             Maps to run: 32****
>
>                          Reduces to run: 1****
>
>                      Block Size (bytes): 1****
>
>                          Bytes to write: 20****
>
>                      Bytes per checksum: 1****
>
>                         Number of files: 500****
>
>                      Replication factor: 1****
>
>              Successful file operations: 16000****
>
> ** **
>
>          # maps that missed the barrier: 0****
>
>                            # exceptions: 0****
>
> ** **
>
>                 TPS: Create/Write/Close: 153****
>
> Avg exec time (ms): Create/Write/Close: 301.6148125****
>
>              Avg Lat (ms): Create/Write: 68.25625****
>
>                     Avg Lat (ms): Close: 233.245****
>
> ** **
>
>                   RAW DATA: AL Total #1: 1092100****
>
>                   RAW DATA: AL Total #2: 3731920****
>
>                RAW DATA: TPS Total (ms): 4825837****
>
>         RAW DATA: Longest Map Time (ms): 208692.0****
>
>                     RAW DATA: Late maps: 0****
>
>               RAW DATA: # of exceptions: 0****
>
> ** **
>
> Hope this helps,****
>
> Ivan****
>
> ** **
>
> *From:* Mark Kerzner [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, June 05, 2013 5:16 PM
> *To:* Hadoop User
> *Subject:* Re: How to test the performance of NN?****
>
> ** **
>
> Hi, Ivan,****
>
> thank you for your willingness to help. First, what do I mean by a more
> efficient NN? - I am experimenting with Kove <http://kove.com/> device,
> persistent memory. I want to do one of the two things:****
>
> 1. Use it for FSDirectory, with the expectation of very fast writes, to
> see if this improves performance.****
>
> 2. Use it for BlockMaps, not for performance, but for fault-tolerance.****
>
> To prepare for this, I want to run some benchmarks on the given install
> just to get experience running it. Then I will run it on a 3-node cluster
> without Kove, and then with Kove.****
>
> Here is what I got from nnbench, and I am trying to find what I should
> watch for to measure NN performance. Does this all make sense?****
>
> Thank you,
> Mark
>
>
> 13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench --------------
> :
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> Version: NameNode Benchmark 0.4
> 13/06/05 16:50:16 INFO hdfs.NNBench:                            Date &
> time: 2013-06-05 16:50:16,412
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench:                         Test
> Operation: open_read
> 13/06/05 16:50:16 INFO hdfs.NNBench:                             Start
> time: 2013-06-05 16:48:34,792
> 13/06/05 16:50:16 INFO hdfs.NNBench:                            Maps to
+
Suresh Srinivas 2013-06-05, 22:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB