Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Testing Hadoop job


Copy link to this message
-
Re: Testing Hadoop job
Hi Nikhil,

MRUnit now supports the 0.20 API as of
https://issues.apache.org/jira/browse/MAPREDUCE-800. There are no plans to
involve partitioners in MRUnit; it is for mappers and reducers only, and not
for full jobs involving input/output formats, partitioners, etc. Use the
LocalJobRunner for that.

See www.cloudera.com/hadoop-mrunit for more (slightly out-of-date)
information.
- Aaron

On Fri, Aug 28, 2009 at 9:15 AM, Nikhil Sawant <[EMAIL PROTECTED]>wrote:

> hi
> thanks Jason, for prompt reply i will go through "pro hadoop". already on
> my to-do list
>
> any idea abt the MRUnit??? has anyone used it?
> i think it is useful as it allows dummy map and reduce drivers which
> accepts (K,V) pair/s and checks the o/p (K,V) pair/s with the expected
> (K,V)......it gives gr8 debugging capabilities while implementing complex
> logics. (all "System.outs" can be seen on console itself!!)
> i have used the basic functionalities of MRUnit testing framework
> i would like to know the limitations (e.g. i found out that MRUnit does not
> check the partioner logic) and its feasibility with hadoop 0.20...
> No proper documentation i found ! :(
>
> cheers
> nikhil
>
>
> Jason Venner wrote:
>
>> I put together a framework for the Pro Hadoop book that I use quite a bit,
>> and has some documentation in the book examples ;)
>> I haven't tried it with 0.20.0 however.
>>
>> The nicest thing that I did with the framework was provide a way to run a
>> persistent mini virtual cluster for running multiple tests on.
>>
>> On Wed, Aug 26, 2009 at 4:50 PM, Nikhil Sawant <[EMAIL PROTECTED]
>> >wrote:
>>
>>
>>
>>> hi
>>>
>>> can u guys suggest some hadoop unit testing framework apart from
>>> MRUnit???
>>> i have used MRUnit but i m not sure abt its feasibilty and support to
>>> hadoop 0.20.....
>>> i could not find a proper documentation for MRUnit, is it available
>>> anywhere?
>>>
>>> --
>>> cheers
>>> nikhil
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB