Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> Mapper Reducer : Unit Test and mocking with static variables


Copy link to this message
-
Re: Mapper Reducer : Unit Test and mocking with static variables
Varene,

You might want to check out MRUnit. It's a unit test harness that contains
mock objects for the context & other associated classes, and works with
JUnit.

It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
Distribution for Hadoop. See
http://archive.cloudera.com/docs/mrunit/index.html for MRUnit documentation.

Cheers,
- Aaron

On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <[EMAIL PROTECTED]> wrote:

> FYI
> I am on Hadoop 0.20.2, JUnit 4
>
>
> Varene Olivier a écrit :
>
>  Hello to all,
>>
>> first of all many thanks for this great piece of software you are all
>> contributing to :)
>>
>> I am actually creating a program in Hadoop MapReduce, but I intend to
>> massively use JUnit Tests to validate my code.
>>
>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>> function with it, but inside my Mapper.map function, I am using Counters
>> to maintain internal statistics. The update of those counters make the test
>> fails with a null pointer (JavaNullPointerException).
>> There is an issue "Counters" of variable visibility from my test ...
>>
>> Do you have any idea how to mock/validate/overcome this issue ?
>>
>>
>>
>> Thanks a lots
>> Best Regards
>> Olivier Varene
>>
>>
>> : Code :
>> --------
>>
>> private static enum Counters {NBLINES};
>> private static IntWritable one = new IntWritable(1);
>> private static LongWritable oneL = new LongWritable(1);
>>
>>
>> // Mapper
>> /**
>>  * outputs 1 for each line encountered
>>  */
>> public static class LCMapper extends
>>    Mapper<LongWritable, Text, IntWritable, LongWritable>
>> {
>>  private final static IntWritable one = new IntWritable(1);
>>  private final static LongWritable oneL = new LongWritable(1);
>>  /**
>>  * for each line encountered outputs 1 for the key 1
>>  */
>>  public void map(LongWritable p_key, Text p_inputs, Context p_context)
>>    throws IOException, InterruptedException
>>  {
>>    p_context.getCounter(Counters.NBLINES).increment(1);
>>    p_context.write(one,oneL);     } // end map
>> } // end Mapper
>>
>> : Test :
>> --------
>>
>> @Test
>> public void testLCMapper() throws IOException, InterruptedException
>> {     LCMapper mapper = new LCMapper();
>>  Text value = new Text("ligneA\nlignB\n");
>>  // mock the map job execution context
>>  Context context = mock(Context.class);
>>  try {
>>    mapper.map(null, value, context);
>>  } catch (NullPointerException e)
>>  {
>>    // exception raised by context.getCounter(...) null pointer
>>    // HOW TO MOCK ???
>>    // Counter is a static enum from the englobing class ...
>>    // issue with variable visibility (from map function) in this test
>>  }
>>  // verify that the Mapper issued (one,oneL) pair
>>  verify(context).write(one,oneL);
>>       } // end testLCMapper
>>
>>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB