Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Re: Errors about MRunit


Copy link to this message
-
Re: Errors about MRunit
Rishi Yadav 2013-04-20, 16:19
your problem is simple, you are mixing mapred (old api) and mapreduce(new
api) libraries. MRUnit has implementation for both apis.

Here's an example of WordCountTest with use of new api.

----------------------------------------------------------------------------------------------------------------------

package com.infoobjects.hadoop.wc;
import java.util.ArrayList;

import java.util.List;
import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mrunit.mapreduce.MapDriver;

import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;

import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;

import org.junit.Before;

import org.junit.Test;
public class WordCountTest {

MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;

ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;

MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable>
mapReduceDriver;
 @Before

public void init() {

 WordMapper mapper = new WordMapper();

 WordReducer reducer = new WordReducer();

 mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();

 mapDriver.setMapper(mapper);

    reduceDriver = ReduceDriver.newReduceDriver(reducer);

    mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);

 }
 @Test

public void testMapper() {

 mapDriver.withInput(new LongWritable(1), new Text("foo bar"));

 mapDriver.withOutput(new Text("foo"), new IntWritable(1));

 mapDriver.withOutput(new Text("bar"), new IntWritable(1));

 mapDriver.runTest();

}

    @Test

  public void testReducer() {

    List<IntWritable> values = new ArrayList<IntWritable>();

    values.add(new IntWritable(1));

    values.add(new IntWritable(1));

    reduceDriver.withInput(new Text("foo"), values);

    reduceDriver.withOutput(new Text("foo"), new IntWritable(2));

    reduceDriver.runTest();

  }

  @Test

  public void testMapReduce() {

   mapReduceDriver.withInput(new LongWritable(1), new Text("brian felix"));

   mapReduceDriver.withOutput(new Text("foo"), new IntWritable(1));

   mapReduceDriver.withOutput(new Text("bar"), new IntWritable(1));

   mapReduceDriver.runTest();

  }
}

Thanks and Regards,

Rishi Yadav

(o) 408.988.2000x113 ||  (f) 408.716.2726

InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*

*INC 500 Fastest growing company in 2012 || 2011*

*Best Place to work in Bay Area 2012 - *SF Business Times and the Silicon
Valley / San Jose Business Journal

2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
On Sat, Apr 20, 2013 at 7:14 AM, 姚吉龙 <[EMAIL PROTECTED]> wrote:

> This is what I got form my eclipse. Why still errors about the lib from
> hadoop
> [image: 内嵌图片 1][image: 内嵌图片 2]
> anybody tell me how to use MRunit and Maven
>
>
> 2013/4/20 Hemanth Yamijala <[EMAIL PROTECTED]>
>
>> Hi,
>>
>> If your goal is to use the new API, I am able to get it to work with the
>> following maven configuration:
>>
>>     <dependency>
>>       <groupId>org.apache.mrunit</groupId>
>>       <artifactId>mrunit</artifactId>
>>       <version>0.9.0-incubating</version>
>>       <classifier>hadoop1</classifier>
>>     </dependency>
>>
>> If I switch with classifier hadoop2, I get the same errors as what you
>> facing.
>>
>> Thanks
>> Hemanth
>>
>>
>> On Sat, Apr 20, 2013 at 3:42 PM, 姚吉龙 <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Everyone
>>>
>>> I am testing my MR programe with MRunit, it's version
>>> is mrunit-0.9.0-incubating-hadoop2. My hadoop version is 1.0.4
>>> The error trace is below:
>>>
>>> java.lang.IncompatibleClassChangeError: Found class
>>> org.apache.hadoop.mapreduce.TaskInputOutputContext, but interface was
>>> expected
>>> at
>>> org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper.createCommon(MockContextWrapper.java:53)
>>>  at
>>> org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.create(MockMapContextWrapper.java:70)
>>> at
>>> org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.<init>(MockMapContextWrapper.java:62)
+
Jagat Singh 2013-04-21, 02:56