Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> A list of some HDFS benchmarks


Copy link to this message
-
Re: A list of some HDFS benchmarks
interesting.

FWIW the work on formally specify (in the Computer Science notion of
"formally") is in HADOOP-9361; the HCFS work being driven by redhat is more
about testing.

Some extra ideas on benchmarking
# something to assess performance of cross FS operations
# it'd be nice to have something that would let you experiment with
different hardware options in that NN
# the gridmix3 MapReduce benchmarks can collect job use stats to generate
synthetic workloads. Maybe we could derive something similar from NN
metrics, so that we could build up a better pool of operations on different
workloads (e.g. HBase, Hive + Tez) and apply them.
# there's work needed on scalability tests across filesystems; for the
'9361 tests I'm making them per-FS programmable for options like max #of
files in a directory test, max filesize etc -any additions there would be
welcome
On 4 September 2013 22:27, Erik Paulson <[EMAIL PROTECTED]> wrote:

> Hello all -
>
> As part of a side project, I've been interested in HDFS benchmarking,
> particularly of the Namenode. To get started, I tried to track down a
> number of different benchmarks and collect a few observations about each.
> I've put together a list here:
>
> http://epaulson.github.io/HadoopInternals/benchmarks.html
>
> The benchmarks I included were:
> DFSIO
> DFSIO-e
> NNBench and NNBenchWithoutMR
> S-Live
> LoadGenerator
> NNThroughputBenchmark
> TestEditLog
> MStress, from Quantcast
> Ohio State Microbenchmarks
> SWIM
>
> (I also wrote a bit about what else I'd like to see in a NN benchmark)
>
> I'd appreciate any corrections, feedback, and pointers to code that I
> missed!
>
> Thanks!
>
> -Erik
>

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB