Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> FW: NNbench and MRBench


Copy link to this message
-
Re: FW: NNbench and MRBench
El 5/7/2011 10:33 PM, [EMAIL PROTECTED] escribió:
> Thanks, Marcos,
>
> Through these links, I still can't find anything about the NNbench and MRBench.
>
> -----Original Message-----
> From: Marcos Ortiz [mailto:[EMAIL PROTECTED]]
> Sent: 2011年5月8日 10:23
> To: [EMAIL PROTECTED]
> Cc: Shi, Stanley
> Subject: Re: FW: NNbench and MRBench
>
> El 5/7/2011 8:53 PM, [EMAIL PROTECTED] escribió:
>    
>> Hi guys,
>>
>> I have a cluster of 16 machines running Hadoop. Now I want to do some benchmark on this cluster with the "nnbench" and "mrbench".
>> I'm new to the hadoop thing and have no one to refer to. I don't know what the supposed result should I have?
>> Now for mrbench, I have an average time of 22sec for a one map job. Is this too bad? What the supposed results might be?
>>
>> For nnbench, what's the supposed results? Below is my result.
>> ===============>>                              Date&   time: 2011-05-05 20:40:25,459
>>
>>                           Test Operation: rename
>>                               Start time: 2011-05-05 20:40:03,820
>>                              Maps to run: 1
>>                           Reduces to run: 1
>>                       Block Size (bytes): 1
>>                           Bytes to write: 0
>>                       Bytes per checksum: 1
>>                          Number of files: 10000
>>                       Replication factor: 1
>>               Successful file operations: 10000
>>
>>           # maps that missed the barrier: 0
>>                             # exceptions: 0
>>
>>                              TPS: Rename: 1763
>>               Avg Exec time (ms): Rename: 0.5672
>>                     Avg Lat (ms): Rename: 0.4844
>> null
>>
>>                    RAW DATA: AL Total #1: 4844
>>                    RAW DATA: AL Total #2: 0
>>                 RAW DATA: TPS Total (ms): 5672
>>          RAW DATA: Longest Map Time (ms): 5672.0
>>                      RAW DATA: Late maps: 0
>>                RAW DATA: # of exceptions: 0
>> ============================>> One more question, when I set maps number to bigger, I get all zeros results:
>> ============================>> Test Operation: create_write
>>                               Start time: 2011-05-03 23:22:39,239
>>                              Maps to run: 160
>>                           Reduces to run: 160
>>                       Block Size (bytes): 1
>>                           Bytes to write: 0
>>                       Bytes per checksum: 1
>>                          Number of files: 1
>>                       Replication factor: 1
>>               Successful file operations: 0
>>
>>           # maps that missed the barrier: 0
>>                             # exceptions: 0
>>
>>                  TPS: Create/Write/Close: 0
>> Avg exec time (ms): Create/Write/Close: 0.0
>>               Avg Lat (ms): Create/Write: NaN
>>                      Avg Lat (ms): Close: NaN
>>
>>                    RAW DATA: AL Total #1: 0
>>                    RAW DATA: AL Total #2: 0
>>                 RAW DATA: TPS Total (ms): 0
>>          RAW DATA: Longest Map Time (ms): 0.0
>>                      RAW DATA: Late maps: 0
>>                RAW DATA: # of exceptions: 0
>> ====================>>
>> Can anyone point me to some documents?
>> I really appreciate your help :)
>>
>> Thanks,
>> stanley
>>
>>      
> You can use these resources:
> http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
> http://answers.oreilly.com/topic/460-how-to-benchmark-a-hadoop-cluster/
> http://wiki.apache.org/hadoop/HardwareBenchmarks
> http://www.quora.com/Apache-Hadoop/Are-there-any-good-Hadoop-benchmark-problems
>
> Regards
>
>    
Well, on the Micheal Noll's post says this:

NameNode benchmark (nnbench)
======================NNBench (see src/test/org/apache/hadoop/hdfs/NNBench.java) is useful for
load testing the NameNode hardware and configuration. It generates a lot
of HDFS-related requests with normally very small "payloads" for the
sole purpose of putting a high HDFS management stress on the NameNode.
The benchmark can simulate requests for creating, reading, renaming and
deleting files on HDFS.

I like to run this test simultaneously from several machines -- e.g.
from a set of DataNode boxes -- in order to hit the NameNode from
multiple locations at the same time.

The syntax of NNBench is as follows:

NameNode Benchmark 0.4
Usage: nnbench <options>
Options:
         -operation <Available operations are create_write open_read
rename delete. This option is mandatory>
          * NOTE: The open_read, rename and delete operations assume
that the files they operate on, are already available. The create_write
operation must be run before running the other operations.
         -maps <number of maps. default is 1. This is not mandatory>
         -reduces <number of reduces. default is 1. This is not mandatory>
         -startTime <time to start, given in seconds from the epoch.
Make sure this is far enough into the future, so all maps (operations)
will start at the same time>. default is launch time + 2 mins. This is
not mandatory
         -blockSize <Block size in bytes. default is 1. This is not
mandatory>
         -bytesToWrite <Bytes to write. default is 0. This is not mandatory>
         -bytesPerChecksum <Bytes per checksum for the files. default is
1. This is not mandatory>
         -numberOfFiles <number of files to create. default is 1. This
is not mandatory>
         -replicationFactorPerFile <Replication factor for the files.
default is 1. This is not mandatory>
         -baseDir <base DFS path. default is /becnhmarks/NNBench. This
is not mandatory>
         -readFileAfterOpen <true or false. if true, it reads the file
and reports the average time to read. This is valid with the open_read
operation. default is false. This is not mandatory>
         -help: Display the help statement

The following comm
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB