Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Breaking down big unit tests


Copy link to this message
-
Re: Breaking down big unit tests
Julien Le Dem 2012-07-20, 01:31
Junit has option to rerun only specific methods in a class, from the IDE or
the command line.
Some of those tests initialize MiniCluster @BeforeClass and turn it off in
the end.
So splitting would possibly increase the runtime.
Tests should be organized by functional unit, so they should be split
accordingly.
Some tests do not require minicluster and would be much faster in Local
Mode. removing Minicluster from the tests which don't need it would speed
them up significantly.
I'm not in favor of splitting arbitrarily in 10 minutes chunks, I am in
favor in splitting when several functional units are tested in the same
class. Also separating MiniCluster based tests from Local Mode would help.

Julien

On Thu, Jul 19, 2012 at 5:14 PM, Thejas Nair <[EMAIL PROTECTED]> wrote:

> We certainly need to look at ways to reduce the runtime of the 'unit'
> tests. Some of them should be migrated to the e2e tests.
>
> But what you want for being able to re-test easily seems to be a way to
> specify specific test case within a Test*.java file . I wonder if junit
> lets you do that.
>
> -Thejas
>
>
>
> On 7/19/12 2:11 PM, Jie Li wrote:
>
>> Hi all,
>>
>> Apparently some unit test classes are so fat that retesting them is a
>> pain.
>> While reducing the full testing time is a long-term goal, shall we just
>> break down those big units into smaller pieces? Here are the running time
>> of top 20 big units :
>>
>> 3,432.68 org.apache.pig.test.**TestEvalPipeline2
>> 2,944.075 org.apache.pig.test.**TestSkewedJoin
>> 1,819.059 org.apache.pig.test.**TestMergeJoin
>> 1,797.877 org.apache.pig.test.TestFRJoin
>> 1,476.097 org.apache.pig.test.**TestEvalPipeline
>> 1,261.661 org.apache.pig.test.**TestFRJoin2
>> 1,164.076 org.apache.pig.test.**TestAccumulator
>> 801.747 org.apache.pig.test.TestBZip
>> 799.689 org.apache.pig.test.TestJoin
>> 792.808 org.apache.pig.test.**TestPigRunner
>> 750.614 org.apache.pig.test.**TestStreaming
>> 743.728 org.apache.pig.test.**TestNativeMapReduce
>> 739.31 org.apache.pig.test.**TestLimitVariable
>> 674.208 org.apache.pig.test.**TestScriptLanguageJavaScript
>> 664.857 org.apache.pig.test.**TestJoinSmoke
>> 653.671 org.apache.pig.test.**TestCounters
>> 621.06 org.apache.pig.test.**TestBestFitCast
>> 541.43 org.apache.pig.test.**TestAlgebraicEval
>> 539.939 org.apache.pig.test.TestGrunt
>>
>> While the full tests take about 10 hours to finish, these top 20 classes
>> account for almost half the time. The idea is to cut them each to
>> 10-minute
>> pieces. Any comment?
>>
>> Jie
>>
>>
>
>