Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Errors about MRunit


+
姚吉龙 2013-04-20, 10:12
+
Hemanth Yamijala 2013-04-20, 10:42
+
Hemanth Yamijala 2013-04-20, 12:04
+
姚吉龙 2013-04-21, 02:17
Copy link to this message
-
Re: Errors about MRunit
Maven is not necessary to use hadoop or MRUnit. That being said I am a big
fan of maven. Please find attached wordcount with maven.

On Sat, Apr 20, 2013 at 7:17 PM, 姚吉龙 <[EMAIL PROTECTED]> wrote:

> Thank for your reply
> But I think the main problem is that I do konw how to use maven with
> hadoop or MRunit
> Any body can give me a example for MRunit project
>
>
>
> 2013/4/21 Rishi Yadav <[EMAIL PROTECTED]>
>
>> your problem is simple, you are mixing mapred (old api) and mapreduce(new
>> api) libraries. MRUnit has implementation for both apis.
>>
>> Here's an example of WordCountTest with use of new api.
>>
>>
>> ----------------------------------------------------------------------------------------------------------------------
>>
>> package com.infoobjects.hadoop.wc;
>>
>>
>> import java.util.ArrayList;
>>
>> import java.util.List;
>>
>>
>> import org.apache.hadoop.io.IntWritable;
>>
>> import org.apache.hadoop.io.LongWritable;
>>
>> import org.apache.hadoop.io.Text;
>>
>> import org.apache.hadoop.mrunit.mapreduce.MapDriver;
>>
>> import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
>>
>> import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
>>
>> import org.junit.Before;
>>
>> import org.junit.Test;
>>
>>
>> public class WordCountTest {
>>
>> MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
>>
>> ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
>>
>> MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable>
>> mapReduceDriver;
>>
>>
>>  @Before
>>
>> public void init() {
>>
>>  WordMapper mapper = new WordMapper();
>>
>>  WordReducer reducer = new WordReducer();
>>
>>  mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
>>
>>  mapDriver.setMapper(mapper);
>>
>>     reduceDriver = ReduceDriver.newReduceDriver(reducer);
>>
>>     mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);
>>
>>  }
>>
>>
>>  @Test
>>
>> public void testMapper() {
>>
>>  mapDriver.withInput(new LongWritable(1), new Text("foo bar"));
>>
>>  mapDriver.withOutput(new Text("foo"), new IntWritable(1));
>>
>>  mapDriver.withOutput(new Text("bar"), new IntWritable(1));
>>
>>  mapDriver.runTest();
>>
>> }
>>
>>     @Test
>>
>>   public void testReducer() {
>>
>>     List<IntWritable> values = new ArrayList<IntWritable>();
>>
>>     values.add(new IntWritable(1));
>>
>>     values.add(new IntWritable(1));
>>
>>     reduceDriver.withInput(new Text("foo"), values);
>>
>>     reduceDriver.withOutput(new Text("foo"), new IntWritable(2));
>>
>>     reduceDriver.runTest();
>>
>>   }
>>
>>
>>
>>   @Test
>>
>>   public void testMapReduce() {
>>
>>    mapReduceDriver.withInput(new LongWritable(1), new Text("brian felix"
>> ));
>>
>>    mapReduceDriver.withOutput(new Text("foo"), new IntWritable(1));
>>
>>    mapReduceDriver.withOutput(new Text("bar"), new IntWritable(1));
>>
>>    mapReduceDriver.runTest();
>>
>>   }
>>
>>
>> }
>>
>> Thanks and Regards,
>>
>> Rishi Yadav
>>
>> (o) 408.988.2000x113 ||  (f) 408.716.2726
>>
>> InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*
>>
>> *INC 500 Fastest growing company in 2012 || 2011*
>>
>> *Best Place to work in Bay Area 2012 - *SF Business Times and the
>> Silicon Valley / San Jose Business Journal
>>
>> 2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
>>
>>
>>
>>
>> On Sat, Apr 20, 2013 at 7:14 AM, 姚吉龙 <[EMAIL PROTECTED]> wrote:
>>
>>> This is what I got form my eclipse. Why still errors about the lib from
>>> hadoop
>>> [image: 内嵌图片 1][image: 内嵌图片 2]
>>> anybody tell me how to use MRunit and Maven
>>>
>>>
>>> 2013/4/20 Hemanth Yamijala <[EMAIL PROTECTED]>
>>>
>>>> Hi,
>>>>
>>>> If your goal is to use the new API, I am able to get it to work with
>>>> the following maven configuration:
>>>>
>>>>     <dependency>
>>>>       <groupId>org.apache.mrunit</groupId>
>>>>       <artifactId>mrunit</artifactId>
>>>>       <version>0.9.0-incubating</version>
>>>>       <classifier>hadoop1</classifier>
+
姚吉龙 2013-04-21, 03:33
+
姚吉龙 2013-04-21, 05:16
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB