Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> Mapper Reducer : Unit Test and mocking with static variables


Copy link to this message
-
Re: Mapper Reducer : Unit Test and mocking with static variables
Varene,

You might want to check out MRUnit. It's a unit test harness that contains
mock objects for the context & other associated classes, and works with
JUnit.

It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
Distribution for Hadoop. See
http://archive.cloudera.com/docs/mrunit/index.html for MRUnit documentation.

Cheers,
- Aaron

On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <[EMAIL PROTECTED]> wrote:

> FYI
> I am on Hadoop 0.20.2, JUnit 4
>
>
> Varene Olivier a écrit :
>
>  Hello to all,
>>
>> first of all many thanks for this great piece of software you are all
>> contributing to :)
>>
>> I am actually creating a program in Hadoop MapReduce, but I intend to
>> massively use JUnit Tests to validate my code.
>>
>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>> function with it, but inside my Mapper.map function, I am using Counters
>> to maintain internal statistics. The update of those counters make the test
>> fails with a null pointer (JavaNullPointerException).
>> There is an issue "Counters" of variable visibility from my test ...
>>
>> Do you have any idea how to mock/validate/overcome this issue ?
>>
>>
>>
>> Thanks a lots
>> Best Regards
>> Olivier Varene
>>
>>
>> : Code :
>> --------
>>
>> private static enum Counters {NBLINES};
>> private static IntWritable one = new IntWritable(1);
>> private static LongWritable oneL = new LongWritable(1);
>>
>>
>> // Mapper
>> /**
>>  * outputs 1 for each line encountered
>>  */
>> public static class LCMapper extends
>>    Mapper<LongWritable, Text, IntWritable, LongWritable>
>> {
>>  private final static IntWritable one = new IntWritable(1);
>>  private final static LongWritable oneL = new LongWritable(1);
>>  /**
>>  * for each line encountered outputs 1 for the key 1
>>  */
>>  public void map(LongWritable p_key, Text p_inputs, Context p_context)
>>    throws IOException, InterruptedException
>>  {
>>    p_context.getCounter(Counters.NBLINES).increment(1);
>>    p_context.write(one,oneL);     } // end map
>> } // end Mapper
>>
>> : Test :
>> --------
>>
>> @Test
>> public void testLCMapper() throws IOException, InterruptedException
>> {     LCMapper mapper = new LCMapper();
>>  Text value = new Text("ligneA\nlignB\n");
>>  // mock the map job execution context
>>  Context context = mock(Context.class);
>>  try {
>>    mapper.map(null, value, context);
>>  } catch (NullPointerException e)
>>  {
>>    // exception raised by context.getCounter(...) null pointer
>>    // HOW TO MOCK ???
>>    // Counter is a static enum from the englobing class ...
>>    // issue with variable visibility (from map function) in this test
>>  }
>>  // verify that the Mapper issued (one,oneL) pair
>>  verify(context).write(one,oneL);
>>       } // end testLCMapper
>>
>>
>>