-Re: Testing Hadoop job
Aaron Kimball 2009-08-29, 00:50
MRUnit now supports the 0.20 API as of
https://issues.apache.org/jira/browse/MAPREDUCE-800. There are no plans to
involve partitioners in MRUnit; it is for mappers and reducers only, and not
for full jobs involving input/output formats, partitioners, etc. Use the
LocalJobRunner for that.
See www.cloudera.com/hadoop-mrunit for more (slightly out-of-date)
On Fri, Aug 28, 2009 at 9:15 AM, Nikhil Sawant <[EMAIL PROTECTED]>wrote:
> thanks Jason, for prompt reply i will go through "pro hadoop". already on
> my to-do list
> any idea abt the MRUnit??? has anyone used it?
> i think it is useful as it allows dummy map and reduce drivers which
> accepts (K,V) pair/s and checks the o/p (K,V) pair/s with the expected
> (K,V)......it gives gr8 debugging capabilities while implementing complex
> logics. (all "System.outs" can be seen on console itself!!)
> i have used the basic functionalities of MRUnit testing framework
> i would like to know the limitations (e.g. i found out that MRUnit does not
> check the partioner logic) and its feasibility with hadoop 0.20...
> No proper documentation i found ! :(
> Jason Venner wrote:
>> I put together a framework for the Pro Hadoop book that I use quite a bit,
>> and has some documentation in the book examples ;)
>> I haven't tried it with 0.20.0 however.
>> The nicest thing that I did with the framework was provide a way to run a
>> persistent mini virtual cluster for running multiple tests on.
>> On Wed, Aug 26, 2009 at 4:50 PM, Nikhil Sawant <[EMAIL PROTECTED]
>>> can u guys suggest some hadoop unit testing framework apart from
>>> i have used MRUnit but i m not sure abt its feasibilty and support to
>>> hadoop 0.20.....
>>> i could not find a proper documentation for MRUnit, is it available