Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> How to test the performance of NN?


Copy link to this message
-
Re: How to test the performance of NN?
Hi, Ivan,

thank you for your willingness to help. First, what do I mean by a more
efficient NN? - I am experimenting with Kove <http://kove.com/> device,
persistent memory. I want to do one of the two things:

1. Use it for FSDirectory, with the expectation of very fast writes, to see
if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.

To prepare for this, I want to run some benchmarks on the given install
just to get experience running it. Then I will run it on a 3-node cluster
without Kove, and then with Kove.

Here is what I got from nnbench, and I am trying to find what I should
watch for to measure NN performance. Does this all make sense?

Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench:
Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench:                            Date &
time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:                         Test
Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench:                             Start
time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench:                            Maps to
run: 12
13/06/05 16:50:16 INFO hdfs.NNBench:                         Reduces to
run: 6
13/06/05 16:50:16 INFO hdfs.NNBench:                     Block Size
(bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench:                         Bytes to
write: 0
13/06/05 16:50:16 INFO hdfs.NNBench:                     Bytes per
checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench:                        Number of
files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench:                     Replication
factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench:             Successful file
operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:         # maps that missed the
barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench:                           #
exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:                         TPS:
Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench:          Avg Exec time (ms):
Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench:                     Avg Lat (ms):
Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench:                   Avg Lat (ms): Read:
NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:                  RAW DATA: AL Total
#1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench:                  RAW DATA: AL Total
#2: 0
13/06/05 16:50:16 INFO hdfs.NNBench:               RAW DATA: TPS Total
(ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench:        RAW DATA: Longest Map Time
(ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench:                    RAW DATA: Late
maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench:              RAW DATA: # of
exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <[EMAIL PROTECTED]> wrote:

>  Hi Mark,****
>
> ** **
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ** **
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ** **
>
> Hope this helps,****
>
> Ivan  ****
>
> ** **
>
> *From:* Mark Kerzner [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ** **
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>