Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Re: Errors about MRunit


+
姚吉龙 2013-04-20, 14:14
+
姚吉龙 2013-04-21, 02:52
Copy link to this message
-
Re: Errors about MRunit
Hello,

Please see this with Maven setup.

https://cwiki.apache.org/confluence/display/MRUNIT/MRUnit+Tutorial

Thanks
On Sun, Apr 21, 2013 at 12:33 PM, Rishi Yadav <[EMAIL PROTECTED]> wrote:

> Maven is not necessary to use hadoop or MRUnit. That being said I am a big
> fan of maven. Please find attached wordcount with maven.
>
>
>
> On Sat, Apr 20, 2013 at 7:17 PM, 姚吉龙 <[EMAIL PROTECTED]> wrote:
>
>> Thank for your reply
>> But I think the main problem is that I do konw how to use maven with
>> hadoop or MRunit
>> Any body can give me a example for MRunit project
>>
>>
>>
>> 2013/4/21 Rishi Yadav <[EMAIL PROTECTED]>
>>
>>> your problem is simple, you are mixing mapred (old api) and
>>> mapreduce(new api) libraries. MRUnit has implementation for both apis.
>>>
>>> Here's an example of WordCountTest with use of new api.
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------------------------
>>>
>>> package com.infoobjects.hadoop.wc;
>>>
>>>
>>> import java.util.ArrayList;
>>>
>>> import java.util.List;
>>>
>>>
>>> import org.apache.hadoop.io.IntWritable;
>>>
>>> import org.apache.hadoop.io.LongWritable;
>>>
>>> import org.apache.hadoop.io.Text;
>>>
>>> import org.apache.hadoop.mrunit.mapreduce.MapDriver;
>>>
>>> import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
>>>
>>> import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
>>>
>>> import org.junit.Before;
>>>
>>> import org.junit.Test;
>>>
>>>
>>> public class WordCountTest {
>>>
>>> MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
>>>
>>> ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
>>>
>>> MapReduceDriver<LongWritable, Text, Text, IntWritable, Text,
>>> IntWritable> mapReduceDriver;
>>>
>>>
>>>  @Before
>>>
>>> public void init() {
>>>
>>>  WordMapper mapper = new WordMapper();
>>>
>>>  WordReducer reducer = new WordReducer();
>>>
>>>  mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
>>>
>>>  mapDriver.setMapper(mapper);
>>>
>>>     reduceDriver = ReduceDriver.newReduceDriver(reducer);
>>>
>>>     mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper,
>>> reducer);
>>>
>>>  }
>>>
>>>
>>>  @Test
>>>
>>> public void testMapper() {
>>>
>>>  mapDriver.withInput(new LongWritable(1), new Text("foo bar"));
>>>
>>>  mapDriver.withOutput(new Text("foo"), new IntWritable(1));
>>>
>>>  mapDriver.withOutput(new Text("bar"), new IntWritable(1));
>>>
>>>  mapDriver.runTest();
>>>
>>> }
>>>
>>>     @Test
>>>
>>>   public void testReducer() {
>>>
>>>     List<IntWritable> values = new ArrayList<IntWritable>();
>>>
>>>     values.add(new IntWritable(1));
>>>
>>>     values.add(new IntWritable(1));
>>>
>>>     reduceDriver.withInput(new Text("foo"), values);
>>>
>>>     reduceDriver.withOutput(new Text("foo"), new IntWritable(2));
>>>
>>>     reduceDriver.runTest();
>>>
>>>   }
>>>
>>>
>>>
>>>   @Test
>>>
>>>   public void testMapReduce() {
>>>
>>>    mapReduceDriver.withInput(new LongWritable(1), new Text("brian felix"
>>> ));
>>>
>>>    mapReduceDriver.withOutput(new Text("foo"), new IntWritable(1));
>>>
>>>    mapReduceDriver.withOutput(new Text("bar"), new IntWritable(1));
>>>
>>>    mapReduceDriver.runTest();
>>>
>>>   }
>>>
>>>
>>> }
>>>
>>> Thanks and Regards,
>>>
>>> Rishi Yadav
>>>
>>> (o) 408.988.2000x113 ||  (f) 408.716.2726
>>>
>>> InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*
>>>
>>> *INC 500 Fastest growing company in 2012 || 2011*
>>>
>>> *Best Place to work in Bay Area 2012 - *SF Business Times and the
>>> Silicon Valley / San Jose Business Journal
>>>
>>> 2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
>>>
>>>
>>>
>>>
>>> On Sat, Apr 20, 2013 at 7:14 AM, 姚吉龙 <[EMAIL PROTECTED]> wrote:
>>>
>>>> This is what I got form my eclipse. Why still errors about the lib from
>>>> hadoop
>>>> [image: 内嵌图片 1][image: 内嵌图片 2]
>>>> anybody tell me how to use MRunit and Maven
>>>>
>>>>
>>>> 2013/4/20 Hemanth Yamijala <[EMAIL PROTECTED]>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB