-Re: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce contrib]
Aaron Kimball 2011-02-17, 19:27
Working to develop code as a client of Hadoop is a path full of landmines.
The more tools we can provide to users to improve the quality of their code,
the better. I think it is important, though, to draw a clear distinction
between tools intended for different audiences. Talking about system testing
tools for Hadoop release/QA processes is good, but one of the benefits I see
of calling MRUnit (designed for client app developers) out of the Hadoop
project at large is to increase its usability. Conflating it with a system
testing tool (for release engineers) would not fulfill that need.
As long as the new project can release several distinct artifacts in a way
that makes their intent clear to the user community, I'm in favor of
gathering as many perspectives on Hadoop testing under one "roof" as
On Wed, Feb 16, 2011 at 10:19 AM, Rottinghuis, Joep
> Having a coherent approach for system level testing increases confidence in
> the various Hadoop releases and and will reduce the effort to take any (set
> of ) changes from development into production.
> The more automated and formalized system testing, the better!
> From: [EMAIL PROTECTED] [[EMAIL PROTECTED]] On Behalf Of Konstantin Boudnik [
> [EMAIL PROTECTED]]
> Sent: Tuesday, February 15, 2011 1:58 PM
> To: [EMAIL PROTECTED]
> Subject: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
> While MrUnit discussion draws to its natural conclusion I would like
> to bring up another point which might be well aligned with that
> discussion. Patrick Hunt has brought up this idea earlier today and I
> believe it has to be elaborated further.
> A number of testing projects both for Hadoop and Hadoop-related
> component were brought to life over last year or two. Among those are
> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
> focusing on more or less the same problem e.g. validation of Hadoop or
> on-top-of-Hadoop components, or application level testing for Hadoop.
> However, the fact that they all are spread across a wide variety of
> projects seems to confuse/mislead Hadoop users.
> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
> project which will take care about development and support of common
> (where's possible) tools, frameworks and the like? Please feel free to
> share your thoughts :)
> Take care,
> Konstantin (Cos) Boudnik
> On Tue, Feb 15, 2011 at 10:44, Eric Sammer <[EMAIL PROTECTED]> wrote:
> > I've started the wiki page proposal for Incubator for mrunit. I'll ping
> > people off list for mentoring. Much appreciated for all the help!
> > On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:
> >> I'm happy to help mentor as well.
> >> Cheers,
> >> Nige
> >> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
> >> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
> >> > <[EMAIL PROTECTED]> wrote:
> >> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
> >> MRUnit, I would be happy to help.
> >> >
> >> > I was going to suggest the same thing (mrunit to incubator). I would
> >> > also be happy to be a mentor.
> >> >
> >> > Patrick