Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Flaky Testcase Detector


Copy link to this message
-
Flaky Testcase Detector
Hey,

Recently I'm working on some hbase test case failures and I think it's
useful if we can see a report of all failed test cases from most recent
runs. So we can easily see how flaky a test case is. I wrote a tool some
time back and below are some reports against different branches from today's
run. You can get the tools' source at
https://github.com/jeffreyz88/jenkins-tools. If we can run it daily and send
out an email then we can quickly notice what may break after recent
check-ins.

Notes:
1) " 873  874  875  876  877  878  879  880  881" are builds which has
failed test cases in current or previous runs
2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"

HBase-0.95 (from last 10 runs configurable)

Failed Test Cases                21   22   23   24   25   27
=======================================================org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
adoop.hbase.catalog.testmetamigrationconvertingtopb    0    0    0    0   -1
-1
org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkd
eletecolumn   -1    0    0    0    0    0
org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache.
hadoop.hbase.coprocessor.example.testrowcountendpoint   -1    0    0    0
0    0
org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver.
org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver
-1    0    0    0    0    0
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock
edritondeadrs    1   -1   -1    0    1    1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul
dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
1    1    1    1    1   -1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd
ownfixupwhendaughterhassplit    1    1    1   -1   -1   -1
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning    0    1
1    1   -1    0
org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplicat
ion    1    1    1    1    1   -1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov
er    1   -1    0    1   -1    0
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q
ueuefailover    0    1   -1    0   -1    0
org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hado
op.hbase.security.access.testaccesscontroller    0   -1    0    0    0    0

As you can see we have a few test cases don't run successfully at all or
recently.

HBase-TRUNK (from last 10 runs)

Failed Test Cases              3908 3909 3910 3912 3913 3914 3915 3916
=======================================================org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
adoop.hbase.catalog.testmetamigrationconvertingtopb    0   -1    0    0    0
0   -1   -1
org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempt
y    1    1    1    1    1    1   -1    0
org.apache.hadoop.hbase.client.testscannertimeout.test3686a    1    1   -1
0    1    1    1    1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnap
shotregionoperationsindependent    0    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnaps
hotregionoperationsindependent    1    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmoveregion
1    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddelete
onfailure    1    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock
edritondeadrs    1    1    1    1    1   -1    0    1
org.apache.hadoop.hbase.master.testtablelockmanager.testdelete    1    1
-1    0    1    1    1    1
org.apache.hadoop.hbase.procedure.testzkprocedurecontrollers.testsimplezkcoh
ortmembercontroller    1    1    1    1    1    1   -1    0
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testexist
ingznodeblockssplitandwerollback    1    1    1    1    1   -1    0    1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul
dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
1    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd
ownfixupwhendaughterhassplit   -1   -1   -1    0   -1    0   -1    0
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning    0   -1
-1    0    1    1    1    1
org.apache.hadoop.hbase.replication.testmultislavereplication.testmultislave
replication    1    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov
er    1    1    1   -1    0   -1    0    1
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q
ueuefailover    1    1    1   -1    0   -1    0    1
org.apache.hadoop.hbase.security.access.testaccesscontrolfilter.testqualifie
raccess   -1    0    1   -1    0    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testdegenerateregions    1   -1
0    1    1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testfixassignmentsandnohdfschecki
ng    1   -1    0    1    1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testfixbytable    1   -1    0
1    1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testhbckthreadpooling    1   -1
0    1    1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testlingeringreferencefile    1
-1    0    1    1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testmissingfirstregion    1   -1
0    1    1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testmissinglastregion    1   -1
0    1    1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinhdfs    1   -1    0    1
1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinmetaordep