HBase, mail # dev - Re: Flaky Testcase Detector

Flaky Testcase Detector

Jeffrey Zhong 2013-03-06, 02:18

Hey,

Recently I'm working on some hbase test case failures and I think it's
useful if we can see a report of all failed test cases from most recent
runs. So we can easily see how flaky a test case is. I wrote a tool some
time back and below are some reports against different branches from today's
run. You can get the tools' source at
https://github.com/jeffreyz88/jenkins-tools. If we can run it daily and send
out an email then we can quickly notice what may break after recent
check-ins.

Notes:
1) " 873 874 875 876 877 878 879 880 881" are builds which has
failed test cases in current or previous runs
2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"

HBase-0.95 (from last 10 runs configurable)

Failed Test Cases 21 22 23 24 25 27
=======================================================org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
adoop.hbase.catalog.testmetamigrationconvertingtopb 0 0 0 0 -1
-1
org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkd
eletecolumn -1 0 0 0 0 0
org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache.
hadoop.hbase.coprocessor.example.testrowcountendpoint -1 0 0 0
0 0
org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver.
org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver
-1 0 0 0 0 0
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock
edritondeadrs 1 -1 -1 0 1 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul
dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
1 1 1 1 1 -1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd
ownfixupwhendaughterhassplit 1 1 1 -1 -1 -1
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 1
1 1 -1 0
org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplicat
ion 1 1 1 1 1 -1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov
er 1 -1 0 1 -1 0
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q
ueuefailover 0 1 -1 0 -1 0
org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hado
op.hbase.security.access.testaccesscontroller 0 -1 0 0 0 0

As you can see we have a few test cases don't run successfully at all or
recently.

HBase-TRUNK (from last 10 runs)

Failed Test Cases 3908 3909 3910 3912 3913 3914 3915 3916
=======================================================org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
adoop.hbase.catalog.testmetamigrationconvertingtopb 0 -1 0 0 0
0 -1 -1
org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempt
y 1 1 1 1 1 1 -1 0
org.apache.hadoop.hbase.client.testscannertimeout.test3686a 1 1 -1
0 1 1 1 1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnap
shotregionoperationsindependent 0 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnaps
hotregionoperationsindependent 1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmoveregion
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddelete
onfailure 1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock
edritondeadrs 1 1 1 1 1 -1 0 1
org.apache.hadoop.hbase.master.testtablelockmanager.testdelete 1 1
-1 0 1 1 1 1
org.apache.hadoop.hbase.procedure.testzkprocedurecontrollers.testsimplezkcoh
ortmembercontroller 1 1 1 1 1 1 -1 0
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testexist
ingznodeblockssplitandwerollback 1 1 1 1 1 -1 0 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul
dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd
ownfixupwhendaughterhassplit -1 -1 -1 0 -1 0 -1 0
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 -1
-1 0 1 1 1 1
org.apache.hadoop.hbase.replication.testmultislavereplication.testmultislave
replication 1 1 -1 0 1 1 1 1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov
er 1 1 1 -1 0 -1 0 1
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q
ueuefailover 1 1 1 -1 0 -1 0 1
org.apache.hadoop.hbase.security.access.testaccesscontrolfilter.testqualifie
raccess -1 0 1 -1 0 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testdegenerateregions 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testfixassignmentsandnohdfschecki
ng 1 -1 0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testfixbytable 1 -1 0
1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testhbckthreadpooling 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testlingeringreferencefile 1
-1 0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testmissingfirstregion 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testmissinglastregion 1 -1
0 1 1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinhdfs 1 -1 0 1
1 1 1 1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinmetaordep

Re: Flaky Testcase Detector

Andrew Purtell 2013-03-06, 03:12

This is handy. Thanks so much for sharing it.

On Wednesday, March 6, 2013, Jeffrey Zhong wrote:

> Hey,
>
> Recently I'm working on some hbase test case failures and I think it's
> useful if we can see a report of all failed test cases from most recent
> runs. So we can easily see how flaky a test case is. I wrote a tool some
> time back and below are some reports against different branches from
> today's
> run. You can get the tools' source at
> https://github.com/jeffreyz88/jenkins-tools. If we can run it daily and
> send
> out an email then we can quickly notice what may break after recent
> check-ins.
>
> Notes:
> 1) " 873 874 875 876 877 878 879 880 881" are builds which has
> failed test cases in current or previous runs
> 2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"
>
> HBase-0.95 (from last 10 runs configurable)
>
> Failed Test Cases 21 22 23 24 25 27
> =======================================================>
> org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
> adoop.hbase.catalog.testmetamigrationconvertingtopb 0 0 0 0
> -1
> -1
>
> org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkd
> eletecolumn -1 0 0 0 0 0
>
> org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache.
> hadoop.hbase.coprocessor.example.testrowcountendpoint -1 0 0 0
> 0 0
>
> org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver.
> org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver
> -1 0 0 0 0 0
>
> org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock
> edritondeadrs 1 -1 -1 0 1 1
>
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul
> dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
> 1 1 1 1 1 -1
>
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd
> ownfixupwhendaughterhassplit 1 1 1 -1 -1 -1
> org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 1
> 1 1 -1 0
>
> org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplicat
> ion 1 1 1 1 1 -1
>
> org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov
> er 1 -1 0 1 -1 0
>
> org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q
> ueuefailover 0 1 -1 0 -1 0
>
> org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hado
> op.hbase.security.access.testaccesscontroller 0 -1 0 0 0 0
>
> As you can see we have a few test cases don't run successfully at all or
> recently.
>
> HBase-TRUNK (from last 10 runs)
>
> Failed Test Cases 3908 3909 3910 3912 3913 3914 3915 3916
> =======================================================>
> org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
> adoop.hbase.catalog.testmetamigrationconvertingtopb 0 -1 0 0
> 0
> 0 -1 -1
>
> org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempt
> y 1 1 1 1 1 1 -1 0
> org.apache.hadoop.hbase.client.testscannertimeout.test3686a 1 1 -1
> 0 1 1 1 1
>
> org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnap
> shotregionoperationsindependent 0 1 -1 0 1 1 1 1
>
> org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnaps
> hotregionoperationsindependent 1 1 -1 0 1 1 1 1
>
> org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmoveregion
> 1 1 -1 0 1 1 1 1
>
> org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddelete
> onfailure 1 1 -1 0 1 1 1 1
>
> org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock

Best regards,

- Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Flaky Testcase Detector

Stack 2013-03-06, 06:43

Nice. Should we check it into ./dev-tools?
Thanks,
St.Ack
On Tue, Mar 5, 2013 at 7:12 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> This is handy. Thanks so much for sharing it.
>
> On Wednesday, March 6, 2013, Jeffrey Zhong wrote:
>
> > Hey,
> >
> > Recently I'm working on some hbase test case failures and I think it's
> > useful if we can see a report of all failed test cases from most recent
> > runs. So we can easily see how flaky a test case is. I wrote a tool some
> > time back and below are some reports against different branches from
> > today's
> > run. You can get the tools' source at
> > https://github.com/jeffreyz88/jenkins-tools. If we can run it daily and
> > send
> > out an email then we can quickly notice what may break after recent
> > check-ins.
> >
> > Notes:
> > 1) " 873 874 875 876 877 878 879 880 881" are builds which has
> > failed test cases in current or previous runs
> > 2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"
> >
> > HBase-0.95 (from last 10 runs configurable)
> >
> > Failed Test Cases 21 22 23 24 25 27
> > =======================================================> >
> >
> org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
> > adoop.hbase.catalog.testmetamigrationconvertingtopb 0 0 0 0
> > -1
> > -1
> >
> >
> org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkd
> > eletecolumn -1 0 0 0 0 0
> >
> >
> org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache.
> > hadoop.hbase.coprocessor.example.testrowcountendpoint -1 0 0 0
> > 0 0
> >
> >
> org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver.
> >
> org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver
> > -1 0 0 0 0 0
> >
> >
> org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock
> > edritondeadrs 1 -1 -1 0 1 1
> >
> >
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul
> >
> dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
> > 1 1 1 1 1 -1
> >
> >
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd
> > ownfixupwhendaughterhassplit 1 1 1 -1 -1 -1
> > org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0
> 1
> > 1 1 -1 0
> >
> >
> org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplicat
> > ion 1 1 1 1 1 -1
> >
> >
> org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov
> > er 1 -1 0 1 -1 0
> >
> >
> org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q
> > ueuefailover 0 1 -1 0 -1 0
> >
> >
> org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hado
> > op.hbase.security.access.testaccesscontroller 0 -1 0 0 0
> 0
> >
> > As you can see we have a few test cases don't run successfully at all or
> > recently.
> >
> > HBase-TRUNK (from last 10 runs)
> >
> > Failed Test Cases 3908 3909 3910 3912 3913 3914 3915 3916
> > =======================================================> >
> >
> org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
> > adoop.hbase.catalog.testmetamigrationconvertingtopb 0 -1 0 0
> > 0
> > 0 -1 -1
> >
> >
> org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempt
> > y 1 1 1 1 1 1 -1 0
> > org.apache.hadoop.hbase.client.testscannertimeout.test3686a 1 1
> -1
> > 0 1 1 1 1
> >
> >
> org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnap
> > shotregionoperationsindependent 0 1 -1 0 1 1 1 1
> >
> >
> org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnaps
> > hotregionoperationsindependent 1 1 -1 0 1 1 1 1

Re: Flaky Testcase Detector

Jeffrey Zhong 2013-03-06, 18:11

I can check it into ./dev-tools unless someone objects.

Thanks,
-Jeffrey

On 3/5/13 10:43 PM, "Stack" <[EMAIL PROTECTED]> wrote:

>Nice. Should we check it into ./dev-tools?
>Thanks,
>St.Ack
>
>
>On Tue, Mar 5, 2013 at 7:12 PM, Andrew Purtell <[EMAIL PROTECTED]>
>wrote:
>
>> This is handy. Thanks so much for sharing it.
>>
>> On Wednesday, March 6, 2013, Jeffrey Zhong wrote:
>>
>> > Hey,
>> >
>> > Recently I'm working on some hbase test case failures and I think it's
>> > useful if we can see a report of all failed test cases from most
>>recent
>> > runs. So we can easily see how flaky a test case is. I wrote a tool
>>some
>> > time back and below are some reports against different branches from
>> > today's
>> > run. You can get the tools' source at
>> > https://github.com/jeffreyz88/jenkins-tools. If we can run it daily
>>and
>> > send
>> > out an email then we can quickly notice what may break after recent
>> > check-ins.
>> >
>> > Notes:
>> > 1) " 873 874 875 876 877 878 879 880 881" are builds which has
>> > failed test cases in current or previous runs
>> > 2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"
>> >
>> > HBase-0.95 (from last 10 runs configurable)
>> >
>> > Failed Test Cases 21 22 23 24 25 27
>> > =======================================================>> >
>> >
>>
>>org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apach
>>e.h
>> > adoop.hbase.catalog.testmetamigrationconvertingtopb 0 0 0
>>0
>> > -1
>> > -1
>> >
>> >
>>
>>org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbu
>>lkd
>> > eletecolumn -1 0 0 0 0 0
>> >
>> >
>>
>>org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apac
>>he.
>> > hadoop.hbase.coprocessor.example.testrowcountendpoint -1 0 0
>> 0
>> > 0 0
>> >
>> >
>>
>>org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserv
>>er.
>> >
>>
>>org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserv
>>er
>> > -1 0 0 0 0 0
>> >
>> >
>>
>>org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithm
>>ock
>> > edritondeadrs 1 -1 -1 0 1 1
>> >
>> >
>>
>>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testsh
>>oul
>> >
>>
>>dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollba
>>ck
>> > 1 1 1 1 1 -1
>> >
>> >
>>
>>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testsh
>>utd
>> > ownfixupwhendaughterhassplit 1 1 1 -1 -1 -1
>> > org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0
>> 1
>> > 1 1 -1 0
>> >
>> >
>>
>>org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicrepli
>>cat
>> > ion 1 1 1 1 1 -1
>> >
>> >
>>
>>org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefai
>>lov
>> > er 1 -1 0 1 -1 0
>> >
>> >
>>
>>org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompresse
>>d.q
>> > ueuefailover 0 1 -1 0 -1 0
>> >
>> >
>>
>>org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.h
>>ado
>> > op.hbase.security.access.testaccesscontroller 0 -1 0 0 0
>> 0
>> >
>> > As you can see we have a few test cases don't run successfully at all
>>or
>> > recently.
>> >
>> > HBase-TRUNK (from last 10 runs)
>> >
>> > Failed Test Cases 3908 3909 3910 3912 3913 3914 3915 3916
>> > =======================================================>> >
>> >
>>
>>org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apach
>>e.h
>> > adoop.hbase.catalog.testmetamigrationconvertingtopb 0 -1 0
>>0
>> > 0
>> > 0 -1 -1
>> >
>> >
>>
>>org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameise
>>mpt
>> > y 1 1 1 1 1 1 -1 0
>> > org.apache.hadoop.hbase.client.testscannertimeout.test3686a 1 1
>> -1

Re: Flaky Testcase Detector

Ted Yu 2013-03-06, 18:46

Jeff:
Can you log a JIRA and attach your tool there ?

Thanks

On Wed, Mar 6, 2013 at 10:11 AM, Jeffrey Zhong <[EMAIL PROTECTED]>wrote:

>
> I can check it into ./dev-tools unless someone objects.
>
> Thanks,
> -Jeffrey
>
> On 3/5/13 10:43 PM, "Stack" <[EMAIL PROTECTED]> wrote:
>
> >Nice. Should we check it into ./dev-tools?
> >Thanks,
> >St.Ack
> >
> >
> >On Tue, Mar 5, 2013 at 7:12 PM, Andrew Purtell <[EMAIL PROTECTED]>
> >wrote:
> >
> >> This is handy. Thanks so much for sharing it.
> >>
> >> On Wednesday, March 6, 2013, Jeffrey Zhong wrote:
> >>
> >> > Hey,
> >> >
> >> > Recently I'm working on some hbase test case failures and I think it's
> >> > useful if we can see a report of all failed test cases from most
> >>recent
> >> > runs. So we can easily see how flaky a test case is. I wrote a tool
> >>some
> >> > time back and below are some reports against different branches from
> >> > today's
> >> > run. You can get the tools' source at
> >> > https://github.com/jeffreyz88/jenkins-tools. If we can run it daily
> >>and
> >> > send
> >> > out an email then we can quickly notice what may break after recent
> >> > check-ins.
> >> >
> >> > Notes:
> >> > 1) " 873 874 875 876 877 878 879 880 881" are builds which has
> >> > failed test cases in current or previous runs
> >> > 2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"
> >> >
> >> > HBase-0.95 (from last 10 runs configurable)
> >> >
> >> > Failed Test Cases 21 22 23 24 25 27
> >> > =======================================================> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apach
> >>e.h
> >> > adoop.hbase.catalog.testmetamigrationconvertingtopb 0 0 0
> >>0
> >> > -1
> >> > -1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbu
> >>lkd
> >> > eletecolumn -1 0 0 0 0 0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apac
> >>he.
> >> > hadoop.hbase.coprocessor.example.testrowcountendpoint -1 0 0
> >> 0
> >> > 0 0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserv
> >>er.
> >> >
> >>
> >>org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserv
> >>er
> >> > -1 0 0 0 0 0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithm
> >>ock
> >> > edritondeadrs 1 -1 -1 0 1 1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testsh
> >>oul
> >> >
> >>
> >>dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollba
> >>ck
> >> > 1 1 1 1 1 -1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testsh
> >>utd
> >> > ownfixupwhendaughterhassplit 1 1 1 -1 -1 -1
> >> > org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0
> >> 1
> >> > 1 1 -1 0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicrepli
> >>cat
> >> > ion 1 1 1 1 1 -1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefai
> >>lov
> >> > er 1 -1 0 1 -1 0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompresse
> >>d.q
> >> > ueuefailover 0 1 -1 0 -1 0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.h
> >>ado
> >> > op.hbase.security.access.testaccesscontroller 0 -1 0 0 0
> >> 0
> >> >
> >> > As you can see we have a few test cases don't run successfully at all
> >>or
> >> > recently.
> >> >
> >> > HBase-TRUNK (from last 10 runs)
> >> >
> >> > Failed Test Cases 3908 3909 3910 3912 3913 3914 3915 3916
> >> > =======================================================> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apach