Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: Testing Hadoop job


Copy link to this message
-
Re: Testing Hadoop job
Aaron Kimball 2009-08-29, 00:50
Hi Nikhil,

MRUnit now supports the 0.20 API as of
https://issues.apache.org/jira/browse/MAPREDUCE-800. There are no plans to
involve partitioners in MRUnit; it is for mappers and reducers only, and not
for full jobs involving input/output formats, partitioners, etc. Use the
LocalJobRunner for that.

See www.cloudera.com/hadoop-mrunit for more (slightly out-of-date)
information.
- Aaron

On Fri, Aug 28, 2009 at 9:15 AM, Nikhil Sawant <[EMAIL PROTECTED]>wrote:

> hi
> thanks Jason, for prompt reply i will go through "pro hadoop". already on
> my to-do list
>
> any idea abt the MRUnit??? has anyone used it?
> i think it is useful as it allows dummy map and reduce drivers which
> accepts (K,V) pair/s and checks the o/p (K,V) pair/s with the expected
> (K,V)......it gives gr8 debugging capabilities while implementing complex
> logics. (all "System.outs" can be seen on console itself!!)
> i have used the basic functionalities of MRUnit testing framework
> i would like to know the limitations (e.g. i found out that MRUnit does not
> check the partioner logic) and its feasibility with hadoop 0.20...
> No proper documentation i found ! :(
>
> cheers
> nikhil
>
>
> Jason Venner wrote:
>
>> I put together a framework for the Pro Hadoop book that I use quite a bit,
>> and has some documentation in the book examples ;)
>> I haven't tried it with 0.20.0 however.
>>
>> The nicest thing that I did with the framework was provide a way to run a
>> persistent mini virtual cluster for running multiple tests on.
>>
>> On Wed, Aug 26, 2009 at 4:50 PM, Nikhil Sawant <[EMAIL PROTECTED]
>> >wrote:
>>
>>
>>
>>> hi
>>>
>>> can u guys suggest some hadoop unit testing framework apart from
>>> MRUnit???
>>> i have used MRUnit but i m not sure abt its feasibilty and support to
>>> hadoop 0.20.....
>>> i could not find a proper documentation for MRUnit, is it available
>>> anywhere?
>>>
>>> --
>>> cheers
>>> nikhil
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>