Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Errors about MRunit


+
姚吉龙 2013-04-20, 10:12
+
Hemanth Yamijala 2013-04-20, 10:42
+
Hemanth Yamijala 2013-04-20, 12:04
+
姚吉龙 2013-04-21, 02:17
Copy link to this message
-
Re: Errors about MRunit
Maven is not necessary to use hadoop or MRUnit. That being said I am a big
fan of maven. Please find attached wordcount with maven.

On Sat, Apr 20, 2013 at 7:17 PM, 姚吉龙 <[EMAIL PROTECTED]> wrote:

> Thank for your reply
> But I think the main problem is that I do konw how to use maven with
> hadoop or MRunit
> Any body can give me a example for MRunit project
>
>
>
> 2013/4/21 Rishi Yadav <[EMAIL PROTECTED]>
>
>> your problem is simple, you are mixing mapred (old api) and mapreduce(new
>> api) libraries. MRUnit has implementation for both apis.
>>
>> Here's an example of WordCountTest with use of new api.
>>
>>
>> ----------------------------------------------------------------------------------------------------------------------
>>
>> package com.infoobjects.hadoop.wc;
>>
>>
>> import java.util.ArrayList;
>>
>> import java.util.List;
>>
>>
>> import org.apache.hadoop.io.IntWritable;
>>
>> import org.apache.hadoop.io.LongWritable;
>>
>> import org.apache.hadoop.io.Text;
>>
>> import org.apache.hadoop.mrunit.mapreduce.MapDriver;
>>
>> import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
>>
>> import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
>>
>> import org.junit.Before;
>>
>> import org.junit.Test;
>>
>>
>> public class WordCountTest {
>>
>> MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
>>
>> ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
>>
>> MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable>
>> mapReduceDriver;
>>
>>
>>  @Before
>>
>> public void init() {
>>
>>  WordMapper mapper = new WordMapper();
>>
>>  WordReducer reducer = new WordReducer();
>>
>>  mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
>>
>>  mapDriver.setMapper(mapper);
>>
>>     reduceDriver = ReduceDriver.newReduceDriver(reducer);
>>
>>     mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);
>>
>>  }
>>
>>
>>  @Test
>>
>> public void testMapper() {
>>
>>  mapDriver.withInput(new LongWritable(1), new Text("foo bar"));
>>
>>  mapDriver.withOutput(new Text("foo"), new IntWritable(1));
>>
>>  mapDriver.withOutput(new Text("bar"), new IntWritable(1));
>>
>>  mapDriver.runTest();
>>
>> }
>>
>>     @Test
>>
>>   public void testReducer() {
>>
>>     List<IntWritable> values = new ArrayList<IntWritable>();
>>
>>     values.add(new IntWritable(1));
>>
>>     values.add(new IntWritable(1));
>>
>>     reduceDriver.withInput(new Text("foo"), values);
>>
>>     reduceDriver.withOutput(new Text("foo"), new IntWritable(2));
>>
>>     reduceDriver.runTest();
>>
>>   }
>>
>>
>>
>>   @Test
>>
>>   public void testMapReduce() {
>>
>>    mapReduceDriver.withInput(new LongWritable(1), new Text("brian felix"
>> ));
>>
>>    mapReduceDriver.withOutput(new Text("foo"), new IntWritable(1));
>>
>>    mapReduceDriver.withOutput(new Text("bar"), new IntWritable(1));
>>
>>    mapReduceDriver.runTest();
>>
>>   }
>>
>>
>> }
>>
>> Thanks and Regards,
>>
>> Rishi Yadav
>>
>> (o) 408.988.2000x113 ||  (f) 408.716.2726
>>
>> InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*
>>
>> *INC 500 Fastest growing company in 2012 || 2011*
>>
>> *Best Place to work in Bay Area 2012 - *SF Business Times and the
>> Silicon Valley / San Jose Business Journal
>>
>> 2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
>>
>>
>>
>>
>> On Sat, Apr 20, 2013 at 7:14 AM, 姚吉龙 <[EMAIL PROTECTED]> wrote:
>>
>>> This is what I got form my eclipse. Why still errors about the lib from
>>> hadoop
>>> [image: 内嵌图片 1][image: 内嵌图片 2]
>>> anybody tell me how to use MRunit and Maven
>>>
>>>
>>> 2013/4/20 Hemanth Yamijala <[EMAIL PROTECTED]>
>>>
>>>> Hi,
>>>>
>>>> If your goal is to use the new API, I am able to get it to work with
>>>> the following maven configuration:
>>>>
>>>>     <dependency>
>>>>       <groupId>org.apache.mrunit</groupId>
>>>>       <artifactId>mrunit</artifactId>
>>>>       <version>0.9.0-incubating</version>
>>>>       <classifier>hadoop1</classifier>
+
姚吉龙 2013-04-21, 03:33
+
姚吉龙 2013-04-21, 05:16