Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # user >> Terasort/Teragen in smokes


Copy link to this message
-
Re: Terasort/Teragen in smokes
On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> hmm ok. now. Thinking about teragen makes me think of benchmarking..
>
>  In the longer term we could add benchmarking jobs to all the submodules not
>  just mapreduce.  For example there are hi bench and ycsb workloads which
>  might be usable or pulled in as bigtop components ... Iff of course
>  benchmarking is in the cards for bigtop?

It indeed is!

I think doing tera-gen/sort so it can be parameterized will provide a good
basis for future benchmarking (as a bit of reflection: I have did simplistic
yet efficient way of benchmarking HDFS and MR a couple years ago, but my
employer back then has never let it go into the open. Go figure...)

And I have a way of building YCSB against a particular version of Hadoop, so I
guess I will have it packaged as a benchmarking test pretty soon.

Cos

> On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>
> > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> >> Hi guys:
> >>
> >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> >>
> >> Any interest in these as an update to TestHadoopExamples in the MapReduce
> >> smokes?
> >>
> >> If so I could patch them in :)
> >
> > Sure! Sounds like a useful addition.
> >
> > Thanks,
> > Roman.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB