Hello,

I am working on porting some features to HBase trunk, and it looks like there are problems with unit tests in the trunk right now. I am getting timeouts just like the one shown below, but the timeout does not always happen in TestDistributedLogSplitting (e.g. I've seen it happen in TestReplication and other classes). What seems strange to me is that a single test timeout terminates the whole test suite.  Also, apparently the continuous integration server does not handle this situation correctly, because the page at https://builds.apache.org/job/HBase-TRUNK/1989/ claims that there are "no failures", which is far from truth. It looks like this started happening at revision 1987, because when I compare https://builds.apache.org/job/HBase-TRUNK/1987/testReport/ and https://builds.apache.org/job/HBase-TRUNK/1986/testReport/, I see that there are  significantly fewer packages in the former (newer) than in the latter (older) test report.

To summarize, I think this raises the following questions:

  *   Could the changes in https://builds.apache.org/job/HBase-TRUNK/1987/ introduce this problem with unit test timeouts, and how can it be fixed?
  *   Should we make the unit test suite resilient to timeouts in individual tests?  It could skip the timed-out test and continue rather than terminate.
  *   How can we fix the continuous integration server so that it does not report "zero failures" in this case where half of the most part of the suite did not even run?

Thanks,
--Mikhail

On Sat, Jun 25, 2011 at 3:15 PM, Apache Jenkins Server <[EMAIL PROTECTED]> wrote:
See <https://builds.apache.org/job/HBase-TRUNK/1989/changes>

Changes:

[tedyu] HBASE-4025  Server startup fails during startup due to failure in loading
           all table descriptors. (Subbu Iyer via Ted Yu)

[tedyu] HBASE-4028  Hmaster crashes caused by splitting log.
  (gaojinchao via Ted Yu)

[tedyu] HBASE-4029  Inappropriate checking of Logging Mode in HRegionServer
           (Akash Ashok via Ted Yu)

[tedyu] HBASE-451   Remove HTableDescriptor from HRegionInfo
           addendum that fixes TestTableMapReduce

[tedyu] HBASE-4020 "testWritesWhileGetting" unit test needs to be fixed.
 Vandana Ayyalasomayajula via Ted Yu

[tedyu] HBASE-4013  Make ZooKeeperListener Abstract (Akash Ashok via Ted Yu)

------------------------------------------
[...truncated 3171 lines...]
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/rest/transform//package-use.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/security//package-use.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/thrift//package-use.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/thrift/generated//package-use.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/util//package-use.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/zookeeper//package-use.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hbase/tmpl/common//package-use.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hbase/tmpl/master//package-use.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hbase/tmpl/regionserver//package-use.html...>
Building index for all the packages and classes...
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/overview-tree.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/index-all.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/deprecated-list.html...>
Building index for all classes...
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/allclasses-frame.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/allclasses-noframe.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/index.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/overview-summary.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/help-doc.html...>
Generating <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/stylesheet.css...>
80 warnings
[WARNING] Javadoc Warnings
[WARNING] <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java>:445: warning - Tag @link: reference not found: Configuration
[WARNING] <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java>:445: warning - Tag @link: reference not found: Configuration
[WARNING] <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java>:445: warning - Tag @link: reference not found: Configuration
[WARNING] <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java>:445: warning - Tag @link: reference not found: Connection
[WARNING] <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java>:445: warning - Tag @link: reference not found: Connection
[WARNING] <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/src/main/java/org/apache/hadoop/hbase/HServerAddress.java>:89: warning - Tag @link: can't find getAddress()#getHostAddress() in java.net.InetSocketAddress
[WARNING] <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/src/main/java/org/apache/hadoop/hbase/HServerInfo.java>:111: warning - Tag @see: can't find getServerName() in org.apache.hadoop.hbase.HSer

>> Should we make the unit test suite resilient to timeouts in individual
tests?  It could skip the timed-out test and continue rather than terminate.
+1 on the above initiative.

On Sun, Jun 26, 2011 at 5:57 PM, Mikhail Bautin <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I am working on porting some features to HBase trunk, and it looks like
> there are problems with unit tests in the trunk right now. I am getting
> timeouts just like the one shown below, but the timeout does not always
> happen in TestDistributedLogSplitting (e.g. I've seen it happen in
> TestReplication and other classes). What seems strange to me is that a
> single test timeout terminates the whole test suite.  Also, apparently the
> continuous integration server does not handle this situation correctly,
> because the page at https://builds.apache.org/job/HBase-TRUNK/1989/ claims
> that there are "no failures", which is far from truth. It looks like this
> started happening at revision 1987, because when I compare
> https://builds.apache.org/job/HBase-TRUNK/1987/testReport/ and
> https://builds.apache.org/job/HBase-TRUNK/1986/testReport/, I see that
> there are  significantly fewer packages in the former (newer) than in the
> latter (older) test report.
>
> To summarize, I think this raises the following questions:
>
>  *   Could the changes in https://builds.apache.org/job/HBase-TRUNK/1987/introduce this problem with unit test timeouts, and how can it be fixed?
>  *   Should we make the unit test suite resilient to timeouts in individual
> tests?  It could skip the timed-out test and continue rather than terminate.
>  *   How can we fix the continuous integration server so that it does not
> report "zero failures" in this case where half of the most part of the suite
> did not even run?
>
> Thanks,
> --Mikhail
>
> On Sat, Jun 25, 2011 at 3:15 PM, Apache Jenkins Server <
> [EMAIL PROTECTED]> wrote:
> See <https://builds.apache.org/job/HBase-TRUNK/1989/changes>
>
> Changes:
>
> [tedyu] HBASE-4025  Server startup fails during startup due to failure in
> loading
>           all table descriptors. (Subbu Iyer via Ted Yu)
>
> [tedyu] HBASE-4028  Hmaster crashes caused by splitting log.
>  (gaojinchao via Ted Yu)
>
> [tedyu] HBASE-4029  Inappropriate checking of Logging Mode in HRegionServer
>           (Akash Ashok via Ted Yu)
>
> [tedyu] HBASE-451   Remove HTableDescriptor from HRegionInfo
>           addendum that fixes TestTableMapReduce
>
> [tedyu] HBASE-4020 "testWritesWhileGetting" unit test needs to be fixed.
>  Vandana Ayyalasomayajula via Ted Yu
>
> [tedyu] HBASE-4013  Make ZooKeeperListener Abstract (Akash Ashok via Ted
> Yu)
>
> ------------------------------------------
> [...truncated 3171 lines...]
> Generating <
> https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/rest/transform//package-use.html..
> .>
> Generating <
> https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/security//package-use.html..
> .>
> Generating <
> https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/thrift//package-use.html..
> .>
> Generating <
> https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/thrift/generated//package-use.html..
> .>
> Generating <
> https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/util//package-use.html..
> .>
> Generating <
> https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/zookeeper//package-use.html..
> .>
> Generating <
> https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hbase/tmpl/common//package-use.html..
> .>
> Generating <
> https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hbase/tmpl/master//package-use.html..
> .>
> Generating <
> https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/site/apidocs/org/apache/hbase/tmpl/regionserver//package-use.html..

On Sun, Jun 26, 2011 at 5:57 PM, Mikhail Bautin <[EMAIL PROTECTED]> wrote:
> I am working on porting some features to HBase trunk, and it looks like there are problems with unit tests in the trunk right now.

Yes.  We're working on getting them fixed.

> What seems strange to me is that a single test timeout terminates the whole test suite.

I'm not sure how to change that.  Here's config. for surefire maven
plugin: http://maven.apache.org/plugins/maven-surefire-plugin/test-mojo.html

Here is comment on the test timeout flag:

forkedProcessTimeoutInSeconds int 2.4 Kill the forked test process
after a certain number of seconds. If set to 0, wait forever for the
process, never timing out.

I suppose if surefire steps in because test has gone on too long, then
it fails the test run.

I messed with the junit timeout annotation -- a few of our tests have
this set -- and if its timeout elapses, tests keep going just failing
this single test.

So it seems as though when the maven surefire plugin has to step in,
we don't keep going.  I don't see a config. to change this (maybe
someone knows?)
> Also, apparently the continuous integration server does not handle this situation correctly, because the page at https://builds.apache.org/job/HBase-TRUNK/1989/ claims that there are "no failures", which is far from truth.

Well, it does report the build as failed.

> To summarize, I think this raises the following questions:
>
>  *   Could the changes in https://builds.apache.org/job/HBase-TRUNK/1987/ introduce this problem with unit test timeouts, and how can it be fixed?

I don't think the commits that went into build 1987 responsible.  I've
seen this behavior in the past (where timeout of a single test fails
the build but doesn't show in the test report).
>  *   Should we make the unit test suite resilient to timeouts in individual tests?  It could skip the timed-out test and continue rather than terminate.

Not all of our tests are junit 4 but for those that are, we can use
the timeout annotation; e.g. add it to these tests that tend to go on
for the > the surefire configured 15 minute timeout so our tests
continue to run.  Let me do this.

>  *   How can we fix the continuous integration server so that it does not report "zero failures" in this case where half of the most part of the suite did not even run?
>

By following the prescription above.

Thanks for raising this issue Mikhail.

St.Ack

My fault.  Ill-formed xml.  Fixed.
St.Ack

On Tue, Jul 26, 2011 at 11:16 AM, Apache Jenkins Server
<[EMAIL PROTECTED]> wrote:
> See <https://builds.apache.org/job/HBase-TRUNK/2051/changes>
>
> Changes:
>
> [stack] HBASE-3845 data loss because lastSeqWritten can miss memstore edits
>
> [stack] HBASE-4140 book: Update our hadoop vendor section
>
> ------------------------------------------
> [...truncated 931 lines...]
> A         src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon
> A         src/main/jamon/org/apache/hbase/tmpl/master
> A         src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon
> A         src/main/jamon/org/apache/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon
> A         src/main/javadoc
> A         src/main/javadoc/org
> A         src/main/javadoc/org/apache
> A         src/main/javadoc/org/apache/hadoop
> A         src/main/javadoc/org/apache/hadoop/hbase
> A         src/main/javadoc/org/apache/hadoop/hbase/thrift
> A         src/main/javadoc/org/apache/hadoop/hbase/thrift/doc-files
> A         src/main/javadoc/org/apache/hadoop/hbase/thrift/doc-files/Hbase.html
> A         src/main/javadoc/org/apache/hadoop/hbase/thrift/doc-files/style.css
> A         src/main/javadoc/org/apache/hadoop/hbase/thrift/doc-files/index.html
> A         src/main/javadoc/org/apache/hadoop/hbase/thrift/package.html
> A         src/main/javadoc/org/apache/hadoop/hbase/io
> A         src/main/javadoc/org/apache/hadoop/hbase/io/hfile
> A         src/main/javadoc/org/apache/hadoop/hbase/io/hfile/package.html
> A         src/main/javadoc/org/apache/hadoop/hbase/replication
> A         src/main/javadoc/org/apache/hadoop/hbase/replication/package.html
> A         src/main/javadoc/org/apache/hadoop/hbase/ipc
> A         src/main/javadoc/org/apache/hadoop/hbase/ipc/package.html
> A         src/main/javadoc/overview.html
> A         src/main/resources
> A         src/main/resources/hbase-webapps
> A         src/main/resources/hbase-webapps/static
> A         src/main/resources/hbase-webapps/static/hbase.css
> AU        src/main/resources/hbase-webapps/static/hbase_logo_med.gif
> A         src/main/resources/hbase-webapps/regionserver
> A         src/main/resources/hbase-webapps/regionserver/regionserver.jsp
> A         src/main/resources/hbase-webapps/regionserver/index.html
> A         src/main/resources/hbase-webapps/master
> A         src/main/resources/hbase-webapps/master/zk.jsp
> A         src/main/resources/hbase-webapps/master/table.jsp
> A         src/main/resources/hbase-webapps/master/master.jsp
> A         src/main/resources/hbase-webapps/master/index.html
> A         src/main/resources/hbase-default.xml
> A         src/main/resources/org
> A         src/main/resources/org/apache
> A         src/main/resources/org/apache/hadoop
> A         src/main/resources/org/apache/hadoop/hbase
> A         src/main/resources/org/apache/hadoop/hbase/thrift
> A         src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift
> A         src/main/resources/org/apache/hadoop/hbase/mapreduce
> A         src/main/resources/org/apache/hadoop/hbase/mapreduce/RowCounter_Counters.properties
> A         src/main/resources/org/apache/hadoop/hbase/mapred
> A         src/main/resources/org/apache/hadoop/hbase/mapred/RowCounter_Counters.properties
> A         src/main/resources/org/apache/hadoop/hbase/rest
> A         src/main/resources/org/apache/hadoop/hbase/rest/protobuf
> A         src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableSchemaMessage.proto
> A         src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ScannerMessage.proto
> A         src/main/resources/org/apache/hadoop/hbase/rest/protobuf/StorageClusterStatusMessage.proto
> A         src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellSetMessage.proto
> A         src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ColumnSchemaMessage.proto
> A         src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellMessage.proto