Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> How to test the performance of NN?


+
Mark Kerzner 2013-06-05, 22:17
+
Ivan Mitic 2013-06-05, 22:32
Copy link to this message
-
Re: How to test the performance of NN?
Hi, Ivan,

thank you for your willingness to help. First, what do I mean by a more
efficient NN? - I am experimenting with Kove <http://kove.com/> device,
persistent memory. I want to do one of the two things:

1. Use it for FSDirectory, with the expectation of very fast writes, to see
if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.

To prepare for this, I want to run some benchmarks on the given install
just to get experience running it. Then I will run it on a 3-node cluster
without Kove, and then with Kove.

Here is what I got from nnbench, and I am trying to find what I should
watch for to measure NN performance. Does this all make sense?

Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench:
Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench:                            Date &
time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:                         Test
Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench:                             Start
time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench:                            Maps to
run: 12
13/06/05 16:50:16 INFO hdfs.NNBench:                         Reduces to
run: 6
13/06/05 16:50:16 INFO hdfs.NNBench:                     Block Size
(bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench:                         Bytes to
write: 0
13/06/05 16:50:16 INFO hdfs.NNBench:                     Bytes per
checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench:                        Number of
files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench:                     Replication
factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench:             Successful file
operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:         # maps that missed the
barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench:                           #
exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:                         TPS:
Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench:          Avg Exec time (ms):
Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench:                     Avg Lat (ms):
Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench:                   Avg Lat (ms): Read:
NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench:                  RAW DATA: AL Total
#1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench:                  RAW DATA: AL Total
#2: 0
13/06/05 16:50:16 INFO hdfs.NNBench:               RAW DATA: TPS Total
(ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench:        RAW DATA: Longest Map Time
(ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench:                    RAW DATA: Late
maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench:              RAW DATA: # of
exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <[EMAIL PROTECTED]> wrote:

>  Hi Mark,****
>
> ** **
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ** **
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ** **
>
> Hope this helps,****
>
> Ivan  ****
>
> ** **
>
> *From:* Mark Kerzner [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ** **
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>
+
Ivan Mitic 2013-06-06, 06:18
+
Mark Kerzner 2013-06-07, 01:41
+
Suresh Srinivas 2013-06-05, 22:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB