The failed TestMasterObserver passes locally for me in 0.94 branch on
multiple runs.
St.Ack

On Thu, Aug 16, 2012 at 2:28 PM, Apache Jenkins Server
<[EMAIL PROTECTED]> wrote:
> See <https://builds.apache.org/job/HBase-0.94/402/changes>
>
> Changes:
>
> [stack] HBASE-6596 Revert HBASE-5022; it undoes HBC.create -- ADDENDUM
>
> ------------------------------------------
> [...truncated 573 lines...]
> Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.162 sec
> Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.11 sec
> Running org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.343 sec
> Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.482 sec
> Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.572 sec
> Running org.apache.hadoop.hbase.zookeeper.TestZKTable
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.271 sec
> Running org.apache.hadoop.hbase.zookeeper.TestZKLeaderManager
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.07 sec
> Running org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.644 sec
> Running org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.747 sec
> Running org.apache.hadoop.hbase.rest.TestSchemaResource
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.686 sec
> Running org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite
> Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.725 sec
> Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 252.666 sec
> Running org.apache.hadoop.hbase.rest.TestGzipFilter
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.036 sec
> Running org.apache.hadoop.hbase.rest.TestRowResource
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.252 sec
> Running org.apache.hadoop.hbase.rest.TestMultiRowResource
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.916 sec
> Running org.apache.hadoop.hbase.rest.client.TestRemoteTable
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.2 sec
> Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.493 sec
> Running org.apache.hadoop.hbase.rest.TestVersionResource
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.598 sec
> Running org.apache.hadoop.hbase.rest.TestStatusResource
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.459 sec
> Running org.apache.hadoop.hbase.rest.TestTableResource
> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.102 sec
> Running org.apache.hadoop.hbase.TestInfoServers
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.044 sec
> Running org.apache.hadoop.hbase.rest.TestScannerResource
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.336 sec
> Running org.apache.hadoop.hbase.monitoring.TestTaskMonitor
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.307 sec
> Running org.apache.hadoop.hbase.rest.TestScannersWithFilters
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.692 sec
> Running org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.255 sec
> Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.126 sec
> Running org.apache.hadoop.hbase.zookeeper.TestZooKeeperACL

This is interesting, because all tests actually passed, yet the test run is declared a BUILD FAILURE.

Hmm... Can't do an 0.94.1RC until I get at least one successful build.
-- Lars

________________________________
 From: Apache Jenkins Server <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Sunday, July 15, 2012 11:02 AM
Subject: Build failed in Jenkins: HBase-0.94 #330
 
See <https://builds.apache.org/job/HBase-0.94/330/>

------------------------------------------
[...truncated 587 lines...]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.745 sec
Running org.apache.hadoop.hbase.TestDrainingServer
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.139 sec
Running org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.293 sec
Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.977 sec
Running org.apache.hadoop.hbase.catalog.TestCatalogTracker
Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.191 sec
Running org.apache.hadoop.hbase.TestClusterBootOrder
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.729 sec
Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.064 sec
Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.584 sec
Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.457 sec
Running org.apache.hadoop.hbase.mapreduce.TestImportExport
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.675 sec
Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.708 sec
Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.375 sec
Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.66 sec
Running org.apache.hadoop.hbase.mapreduce.TestWALPlayer
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.544 sec
Running org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.94 sec
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.238 sec
Running org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.489 sec
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.14 sec
Running org.apache.hadoop.hbase.regionserver.TestStore
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.318 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLogBench
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.108 sec
Running org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.559 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.338 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.888 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLog
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.421 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.626 sec
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 281.352 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.292 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 247.346 sec
Running org.apache.hadoop.hbase.regionserver.TestCacheOnWriteInSchema
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.218 sec
Running org.apache.hadoop.hbase.regionserver.TestCompactionState
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.59 sec
Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 491.98 sec
Running org.apache.hadoop.hbase.regionserver.TestAtomicOperation
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.298 sec
Running org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.064 sec
Running org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.031 sec
Running org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.944 sec
Running org.apache.hadoop.hbase.regionserver.TestParallelPut
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.575 sec
Running org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.356 sec
Running org.apache.hadoop.hbase.regionserver.TestRpcMetrics
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.895 sec
Running org.apache.hadoop.hbase.regionserver.TestSeekOptimizations
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.481 sec
Running org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler
Running org.apache.hadoop.hbase.r

One possibility was that one or more test(s) had hanging surefire JVM after
printing test pass message.

We need access to the Jenkins machine to fully understand.

Cheers

On Sun, Jul 15, 2012 at 12:26 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> This is interesting, because all tests actually passed, yet the test run
> is declared a BUILD FAILURE.
>
> Hmm... Can't do an 0.94.1RC until I get at least one successful build.
>
>
> -- Lars
>
>
>
> ________________________________
>  From: Apache Jenkins Server <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Sunday, July 15, 2012 11:02 AM
> Subject: Build failed in Jenkins: HBase-0.94 #330
>
> See <https://builds.apache.org/job/HBase-0.94/330/>
>
> ------------------------------------------
> [...truncated 587 lines...]
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.745 sec
> Running org.apache.hadoop.hbase.TestDrainingServer
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.139 sec
> Running org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.293 sec
> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.977 sec
> Running org.apache.hadoop.hbase.catalog.TestCatalogTracker
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.191 sec
> Running org.apache.hadoop.hbase.TestClusterBootOrder
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.729 sec
> Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.064 sec
> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.584 sec
> Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.457 sec
> Running org.apache.hadoop.hbase.mapreduce.TestImportExport
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.675 sec
> Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.708 sec
> Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.375 sec
> Running
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.66 sec
> Running org.apache.hadoop.hbase.mapreduce.TestWALPlayer
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.544 sec
> Running org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.94 sec
> Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.238 sec
> Running org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.489 sec
> Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.14 sec
> Running org.apache.hadoop.hbase.regionserver.TestStore
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.318 sec
> Running org.apache.hadoop.hbase.regionserver.wal.TestHLogBench
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.108 sec
> Running org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.559 sec
> Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.338 sec
> Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.888 sec

It happened again, and two more runs with test failures.
In seems that unless we can accept a build not created by the build machines, but instead locally on my machine, we're at the moment unable to produce a release.
I can easily do that if people are willing to accept an artifact created that way. Please let me know.
I would also support the stance that we should not release another version before we sorted that out. Opinions are welcome.
It's interesting that the security build has none of the problems. There at least 1/2 of the builds succeed.
Do you know how to get access to the Jenkins machines?

What we need to do is (1) analyze the tests and see why the fail/hang and (2) what the heck is up with the Jenkins machines.

Jean-Marc said many tests fail as part of a bigger run and than pass when run in isolation. That is still not acceptable, IMHO.

-- Lars
________________________________

From: Ted Yu <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Sent: Sunday, July 15, 2012 1:07 PM
Subject: Re: Build failed in Jenkins: HBase-0.94 #330

One possibility was that one or more test(s) had hanging surefire JVM after
printing test pass message.

We need access to the Jenkins machine to fully understand.

Cheers

On Sun, Jul 15, 2012 at 12:26 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> This is interesting, because all tests actually passed, yet the test run
> is declared a BUILD FAILURE.
>
> Hmm... Can't do an 0.94.1RC until I get at least one successful build.
>
>
> -- Lars
>
>
>
> ________________________________
>  From: Apache Jenkins Server <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Sunday, July 15, 2012 11:02 AM
> Subject: Build failed in Jenkins: HBase-0.94 #330
>
> See <https://builds.apache.org/job/HBase-0.94/330/>
>
> ------------------------------------------
> [...truncated 587 lines...]
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.745 sec
> Running org.apache.hadoop.hbase.TestDrainingServer
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.139 sec
> Running org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.293 sec
> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.977 sec
> Running org.apache.hadoop.hbase.catalog.TestCatalogTracker
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.191 sec
> Running org.apache.hadoop.hbase.TestClusterBootOrder
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.729 sec
> Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.064 sec
> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.584 sec
> Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.457 sec
> Running org.apache.hadoop.hbase.mapreduce.TestImportExport
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.675 sec
> Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.708 sec
> Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.375 sec
> Running
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.66 sec
> Running org.apache.hadoop.hbase.mapreduce.TestWALPlayer
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.544 sec
> Running org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.94 sec
> Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce

Hi Lars,

What you describe below sounds like an interaction between a hung test from earlier in the build interacting with a later test to cause failure(s). That -Psecurity builds fail less often suggests a different ordering of unit tests produces a different result, and is consistent with that.

I can put a fleet of EC2 workers to running each unit test separately 100 times to find those that might hang or flap. Results would be particular to RHEL6. Will get started tomorrow morning. I'll come back with a list of any I find. We can triage then based on results.

    - Andy
On Jul 15, 2012, at 9:09 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> It happened again, and two more runs with test failures.
> In seems that unless we can accept a build not created by the build machines, but instead locally on my machine, we're at the moment unable to produce a release.
> I can easily do that if people are willing to accept an artifact created that way. Please let me know.
> I would also support the stance that we should not release another version before we sorted that out. Opinions are welcome.
>
>
> It's interesting that the security build has none of the problems. There at least 1/2 of the builds succeed.
>
>
> Do you know how to get access to the Jenkins machines?
>
> What we need to do is (1) analyze the tests and see why the fail/hang and (2) what the heck is up with the Jenkins machines.
>
> Jean-Marc said many tests fail as part of a bigger run and than pass when run in isolation. That is still not acceptable, IMHO.
>
> -- Lars
> ________________________________
>
> From: Ted Yu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
> Sent: Sunday, July 15, 2012 1:07 PM
> Subject: Re: Build failed in Jenkins: HBase-0.94 #330
>
> One possibility was that one or more test(s) had hanging surefire JVM after
> printing test pass message.
>
> We need access to the Jenkins machine to fully understand.
>
> Cheers
>
> On Sun, Jul 15, 2012 at 12:26 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>
>> This is interesting, because all tests actually passed, yet the test run
>> is declared a BUILD FAILURE.
>>
>> Hmm... Can't do an 0.94.1RC until I get at least one successful build.
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>>   From: Apache Jenkins Server <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Sunday, July 15, 2012 11:02 AM
>> Subject: Build failed in Jenkins: HBase-0.94 #330
>>
>> See <https://builds.apache.org/job/HBase-0.94/330/>
>>
>> ------------------------------------------
>> [...truncated 587 lines...]
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.745 sec
>> Running org.apache.hadoop.hbase.TestDrainingServer
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.139 sec
>> Running org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
>> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.293 sec
>> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.977 sec
>> Running org.apache.hadoop.hbase.catalog.TestCatalogTracker
>> Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.191 sec
>> Running org.apache.hadoop.hbase.TestClusterBootOrder
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.729 sec
>> Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.064 sec
>> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.584 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
>> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.457 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestImportExport
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.675 sec

Awesome. Thanks Andy!

----- Original Message -----
From: Andrew Purtell <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Sunday, July 15, 2012 10:22 PM
Subject: Re: Build failed in Jenkins: HBase-0.94 #330

Hi Lars,

What you describe below sounds like an interaction between a hung test from earlier in the build interacting with a later test to cause failure(s). That -Psecurity builds fail less often suggests a different ordering of unit tests produces a different result, and is consistent with that.

I can put a fleet of EC2 workers to running each unit test separately 100 times to find those that might hang or flap. Results would be particular to RHEL6. Will get started tomorrow morning. I'll come back with a list of any I find. We can triage then based on results.

    - Andy
On Jul 15, 2012, at 9:09 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> It happened again, and two more runs with test failures.
> In seems that unless we can accept a build not created by the build machines, but instead locally on my machine, we're at the moment unable to produce a release.
> I can easily do that if people are willing to accept an artifact created that way. Please let me know.
> I would also support the stance that we should not release another version before we sorted that out. Opinions are welcome.
>
>
> It's interesting that the security build has none of the problems. There at least 1/2 of the builds succeed.
>
>
> Do you know how to get access to the Jenkins machines?
>
> What we need to do is (1) analyze the tests and see why the fail/hang and (2) what the heck is up with the Jenkins machines.
>
> Jean-Marc said many tests fail as part of a bigger run and than pass when run in isolation. That is still not acceptable, IMHO.
>
> -- Lars
> ________________________________
>
> From: Ted Yu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
> Sent: Sunday, July 15, 2012 1:07 PM
> Subject: Re: Build failed in Jenkins: HBase-0.94 #330
>
> One possibility was that one or more test(s) had hanging surefire JVM after
> printing test pass message.
>
> We need access to the Jenkins machine to fully understand.
>
> Cheers
>
> On Sun, Jul 15, 2012 at 12:26 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>
>> This is interesting, because all tests actually passed, yet the test run
>> is declared a BUILD FAILURE.
>>
>> Hmm... Can't do an 0.94.1RC until I get at least one successful build.
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>>   From: Apache Jenkins Server <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Sunday, July 15, 2012 11:02 AM
>> Subject: Build failed in Jenkins: HBase-0.94 #330
>>
>> See <https://builds.apache.org/job/HBase-0.94/330/>
>>
>> ------------------------------------------
>> [...truncated 587 lines...]
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.745 sec
>> Running org.apache.hadoop.hbase.TestDrainingServer
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.139 sec
>> Running org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
>> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.293 sec
>> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.977 sec
>> Running org.apache.hadoop.hbase.catalog.TestCatalogTracker
>> Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.191 sec
>> Running org.apache.hadoop.hbase.TestClusterBootOrder
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.729 sec
>> Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.064 sec
>> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.584 sec

On Mon, Jul 16, 2012 at 6:09 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> It happened again, and two more runs with test failures.
> In seems that unless we can accept a build not created by the build machines, but instead locally on my machine, we're at the moment unable to produce a release.

I used to always make releases by building locally.  Its only in the
last six months or year that I figured I could take what was made by
jenkins.
> I can easily do that if people are willing to accept an artifact created that way. Please let me know.
> I would also support the stance that we should not release another version before we sorted that out. Opinions are welcome.
>

I think it'd be fine making the 0.94.1RC0.

> It's interesting that the security build has none of the problems. There at least 1/2 of the builds succeed.
>

Yeah.  Thats interesting.  As Andrew speculates, its different order
of unit tests or extra friction makes tests pass.

>
> Do you know how to get access to the Jenkins machines?
>

It seemed "difficult" last time I asked:
https://issues.apache.org/jira/browse/INFRA-4055

> What we need to do is (1) analyze the tests and see why the fail/hang and (2) what the heck is up with the Jenkins machines.
>

On 1., yeah, its an ongoing job that we've been slacking on.  We could
organize a swat team to dig in.    It'll take a bit of work.  Seems
like a bunch of flakey tests and then there are the times it passes
where a test is hung (but it looks like all passed).

> Jean-Marc said many tests fail as part of a bigger run and than pass when run in isolation. That is still not acceptable, IMHO.
>

Agree.

St.Ack

Thanks Stack,

if nobody disagrees I'll do that for 0.94.1RC0.

There were some test runs that had no failed tests (the build then failed for other reasons),
so presumably the test code is flaky and the current situation is no indication of bad production code.
Thanks.

-- Lars
----- Original Message -----
From: Stack <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Cc:
Sent: Monday, July 16, 2012 2:46 PM
Subject: Re: Build failed in Jenkins: HBase-0.94 #330

On Mon, Jul 16, 2012 at 6:09 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> It happened again, and two more runs with test failures.
> In seems that unless we can accept a build not created by the build machines, but instead locally on my machine, we're at the moment unable to produce a release.

I used to always make releases by building locally.  Its only in the
last six months or year that I figured I could take what was made by
jenkins.
> I can easily do that if people are willing to accept an artifact created that way. Please let me know.
> I would also support the stance that we should not release another version before we sorted that out. Opinions are welcome.
>

I think it'd be fine making the 0.94.1RC0.

> It's interesting that the security build has none of the problems. There at least 1/2 of the builds succeed.
>

Yeah.  Thats interesting.  As Andrew speculates, its different order
of unit tests or extra friction makes tests pass.

>
> Do you know how to get access to the Jenkins machines?
>

It seemed "difficult" last time I asked:
https://issues.apache.org/jira/browse/INFRA-4055

> What we need to do is (1) analyze the tests and see why the fail/hang and (2) what the heck is up with the Jenkins machines.
>

On 1., yeah, its an ongoing job that we've been slacking on.  We could
organize a swat team to dig in.    It'll take a bit of work.  Seems
like a bunch of flakey tests and then there are the times it passes
where a test is hung (but it looks like all passed).

> Jean-Marc said many tests fail as part of a bigger run and than pass when run in isolation. That is still not acceptable, IMHO.
>

Agree.

St.Ack

On Mon, Jul 16, 2012 at 3:28 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> On 1., yeah, its an ongoing job that we've been slacking on.  We could
> organize a swat team to dig in.    It'll take a bit of work.  Seems
> like a bunch of flakey tests and then there are the times it passes
> where a test is hung (but it looks like all passed).

I'm on the hunt for flaky tests.

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)

I dispatched each unit test individually to 20 EC2 c1.mediums (64 bit
system, 2 VCPUs, kind of slow on purpose but still allowing some
thread concurrency). On the instance each test was run 100 times or
until failure. For each iteration after Maven exited the process table
was checked to see if any surefire processes lingered, and if so the
test would also be reported failed.

OS: Amazon Linux AMI release 2012.03
uname: Linux 3.2.21-1.32.6.amzn1.x86_64 #1 SMP Sat Jun 23 02:32:15 UTC
2012 x86_64 x86_64 x86_64 GNU/Linux
JVM: java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.3)
(amazon-52.1.11.3.45.amzn1-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

Here are the tests that failed to complete successful runs in the above:

TestCatalogTracker
    hangs on a join in testServerNotRunningIOException waiting on a CT
that is stuck on CatalogTracker.waitForMeta and will linger in the
background

TestColumnSeeking
    testDuplicateVersions(org.apache.hadoop.hbase.regionserver.TestColumnSeeking):
expected:<0> but was:<200>

TestAtomicOperation
    testMultiRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation):
expected:<0> but was:<1>

TestSplitLogManager
    testOrphanTaskAcquisition(org.apache.hadoop.hbase.master.TestSplitLogManager):
java.lang.AssertionError

TestRegionRebalancing
    testRebalanceOnRegionServerNumberChange(org.apache.hadoop.hbase.TestRegionRebalancing):
After 5 attempts, region assignments were not balanced.

TestDrainingServer
    junit.framework.AssertionFailedError from
org.apache.hadoop.hbase.TestDrainingServer.setUpBeforeClass

TestMasterObserver
    testTableOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver):
org.apache.hadoop.hbase.InvalidFamilyOperationException: Column family
'fam2' does not exist
    testRegionTransitionOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver):
org.apache.hadoop.hbase.TableExistsException: observed_table

TestServerCustomProtocol
    testSingleMethod(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol):
Results should contain region
test,bbb,1342509423473.2c0326188f899f3e91ec5eb623959c13. for row 'bbb'

TestFromClientSide
    testPoolBehavior(org.apache.hadoop.hbase.client.TestFromClientSide):
expected:<3> but was:<4>

TestZooKeeper
    testClientSessionExpired(org.apache.hadoop.hbase.TestZooKeeper)

TestReplication
    testDisableInactivePeer(org.apache.hadoop.hbase.replication.TestReplication):
Shutting down

TestMasterReplication
    testSimplePutDelete(org.apache.hadoop.hbase.replication.TestMasterReplication):
Waited too much time for put replication

TestMultiSlaveReplication
    testMultiSlaveReplication(org.apache.hadoop.hbase.replication.TestMultiSlaveReplication):
Unable to add peer

TestReplicationPeer
    testResetZooKeeperSession(org.apache.hadoop.hbase.replication.TestReplicationPeer):
ReplicationPeer ZooKeeper session was not properly expired.

I didn't get to them all before AWS yanked back my spot instances but
I ordered the list from most likely to least, the remaining tests were
in io.hfile.*, thrift.*, and util.*

I'll circle back, confirm each individually, and open JIRAs with more detail.

The cluster of replication test failures are a concern, but I've seen
in other environments such as this one that the tests are timing
dependent. On a slow or busy test system they can fail with "waited
too much time ...". So a solution for this is to not use the system
clock but instead EnvironmentEdge or whatever incremented only when
the test process has CPU time. I haven't looked into this in detail
yet.

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)

Thanks for doing this Andy.
I looked through the 0.94 Jenkins runs still available and found the following tests failing:

6 org.apache.hadoop.hbase.TestZooKeeper.testClientSessionExpired
5 org.apache.hadoop.hbase.replication.TestReplicationPeer.testResetZooKeeperSession
2 org.apache.hadoop.hbase.replication.TestReplication.queueFailover
2 org.apache.hadoop.hbase.client.TestAdmin.testEnableTableRoundRobinAssignment
2 org.apache.hadoop.hbase.TestDrainingServer.org.apache.hadoop.hbase.TestDrainingServer
1 org.apache.hadoop.hbase.client.TestAdmin.testOnlineChangeTableSchema
1 org.apache.hadoop.hbase.client.TestShell.testRunShellTests
1 org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testRegionTransitionOperations
1 org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations
1 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache.testBackgroundEvictionThread[1]
1 org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeMinorCompaction
1 org.apache.hadoop.hbase.master.TestAssignmentManager.testBalanceOnMasterFailoverScenarioWithClosedNode
1 org.apache.hadoop.hbase.master.TestAssignmentManager.testBalanceOnMasterFailoverScenarioWithOfflineNode
1 org.apache.hadoop.hbase.master.TestSplitLogManager.testVanishingTaskZNode
1 org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.testRowRange
1 org.apache.hadoop.hbase.regionserver.TestSplitLogWorker.testAcquireTaskAtStartup
1 org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testShutdownFixupWhenDaughterHasSplit
1 org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK
1 org.apache.hadoop.hbase.util.TestFSUtils.testcomputeHDFSBlocksDistribution
1 org.apache.hadoop.hbase.util.TestHBaseFsck.testNotInMetaHole
1 org.apache.hadoop.hbase.util.TestHBaseFsck.testNotInMetaOrDeployedHole
1 org.apache.hadoop.hbase.util.TestHBaseFsck.testOverlapAndOrphan
I guess the takeaway is that there is not silver bullet here, but that'd we'd get some mileage by fixing or disabling these two:
org.apache.hadoop.hbase.TestZooKeeper.testClientSessionExpired
org.apache.hadoop.hbase.replication.TestReplicationPeer.testResetZooKeeperSession

They account for 11 of 34 failed runs.

I'll file a jira for me to look at these.

-- Lars
----- Original Message -----
From: Andrew Purtell <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc: Lars Hofhansl <[EMAIL PROTECTED]>; michael stack <[EMAIL PROTECTED]>
Sent: Tuesday, July 17, 2012 10:08 AM
Subject: Re: Build failed in Jenkins: HBase-0.94 #330

I dispatched each unit test individually to 20 EC2 c1.mediums (64 bit
system, 2 VCPUs, kind of slow on purpose but still allowing some
thread concurrency). On the instance each test was run 100 times or
until failure. For each iteration after Maven exited the process table
was checked to see if any surefire processes lingered, and if so the
test would also be reported failed.

OS: Amazon Linux AMI release 2012.03
uname: Linux 3.2.21-1.32.6.amzn1.x86_64 #1 SMP Sat Jun 23 02:32:15 UTC
2012 x86_64 x86_64 x86_64 GNU/Linux
JVM: java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.3)
(amazon-52.1.11.3.45.amzn1-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

Here are the tests that failed to complete successful runs in the above:

TestCatalogTracker
    hangs on a join in testServerNotRunningIOException waiting on a CT
that is stuck on CatalogTracker.waitForMeta and will linger in the
background

TestColumnSeeking
    testDuplicateVersions(org.apache.hadoop.hbase.regionserver.TestColumnSeeking):
expected:<0> but was:<200>

TestAtomicOperation
    testMultiRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation):
expected:<0> but was:<1>

TestSplitLogManager
    testOrphanTaskAcquisition(org.apache.hadoop.hbase.master.TestSplitLogManager):
java.lang.AssertionError

TestRegionRebalancing
    testRebalanceOnRegionServerNumberChange(org.apache.hadoop.hbase.TestRegionRebalancing):
After 5 attempts, region assignments were not balanced.

TestDrainingServer
    junit.framework.AssertionFailedError from
org.apache.hadoop.hbase.TestDrainingServer.setUpBeforeClass

TestMasterObserver
    testTableOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver):
org.apache.hadoop.hbase.InvalidFamilyOperationException: Column family
'fam2' does not exist
    testRegionTransitionOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver):
org.apache.hadoop.hbase.TableExistsException: observed_table

TestServerCustomProtocol
    testSingleMethod(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol):
Results should contain region
test,bbb,1342509423473.2c0326188f899f3e91ec5eb623959c13. for row 'bbb'

TestFromClientSide
    testPoolBehavior(org.apache.hadoop.hbase.client.TestFromClientSide):
expected:<3> but was:<4>

TestZooKeeper
    testClientSessionExpired(org.apache.hadoop.hbase.TestZooKeeper)

TestReplication
    testDisableInactivePeer(org.apache.hadoop.hbase.replication.TestReplication):
Shutting down

TestMasterReplication
    testSimplePutDelete(org.apache.hadoop.hbase.replication.TestMasterReplication):
Waited too much time for put replication

TestMultiSlaveReplication
    testMultiSlaveReplication(org.apache.hadoop.hbase.replication.TestMultiSlaveReplication):
Unable to add peer

TestReplicationPeer
    testResetZooKeeperSession(org.apache.hadoop.hbase.replication.TestReplicationPeer):
ReplicationPeer ZooKeeper session was not properly expired.

I didn't get to them all before AWS yanked back my spot instances but
I ordered the list from most likely to least, the remaining tests were
in io.hfile.*, thrift.*, and util.*

I'll circle back, confirm each individually, and open JIRAs with more detail.

The cluster of replication test failures are a concern, but I've seen
in other environments such as this one that the tests are timing
dependent. On a slow or busy test system they can fail with "waited
too much time ...". So a so

On Tue, Jul 17, 2012 at 2:15 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> 1 org.apache.hadoop.hbase.client.TestAdmin.testOnlineChangeTableSchema

This one is interesting, because a few tests like TestAdmin,
TestFromClientSide, etc. I actually ran twice (i.e. 200 times) out of
an abundance of suspicion. TestAdmin passed for me. Maybe 2 VCPU had
insufficient parallelism.

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)

That TestMultiSlaveReplication doesn't pass means something pretty
fundamental was broken...

J-D

On Tue, Jul 17, 2012 at 10:08 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> I dispatched each unit test individually to 20 EC2 c1.mediums (64 bit
> system, 2 VCPUs, kind of slow on purpose but still allowing some
> thread concurrency). On the instance each test was run 100 times or
> until failure. For each iteration after Maven exited the process table
> was checked to see if any surefire processes lingered, and if so the
> test would also be reported failed.
>
> OS: Amazon Linux AMI release 2012.03
> uname: Linux 3.2.21-1.32.6.amzn1.x86_64 #1 SMP Sat Jun 23 02:32:15 UTC
> 2012 x86_64 x86_64 x86_64 GNU/Linux
> JVM: java version "1.6.0_24"
>     OpenJDK Runtime Environment (IcedTea6 1.11.3)
> (amazon-52.1.11.3.45.amzn1-x86_64)
>     OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
>
> Here are the tests that failed to complete successful runs in the above:
>
> TestCatalogTracker
>     hangs on a join in testServerNotRunningIOException waiting on a CT
> that is stuck on CatalogTracker.waitForMeta and will linger in the
> background
>
> TestColumnSeeking
>     testDuplicateVersions(org.apache.hadoop.hbase.regionserver.TestColumnSeeking):
> expected:<0> but was:<200>
>
> TestAtomicOperation
>     testMultiRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation):
> expected:<0> but was:<1>
>
> TestSplitLogManager
>     testOrphanTaskAcquisition(org.apache.hadoop.hbase.master.TestSplitLogManager):
> java.lang.AssertionError
>
> TestRegionRebalancing
>     testRebalanceOnRegionServerNumberChange(org.apache.hadoop.hbase.TestRegionRebalancing):
> After 5 attempts, region assignments were not balanced.
>
> TestDrainingServer
>     junit.framework.AssertionFailedError from
> org.apache.hadoop.hbase.TestDrainingServer.setUpBeforeClass
>
> TestMasterObserver
>     testTableOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver):
> org.apache.hadoop.hbase.InvalidFamilyOperationException: Column family
> 'fam2' does not exist
>     testRegionTransitionOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver):
> org.apache.hadoop.hbase.TableExistsException: observed_table
>
> TestServerCustomProtocol
>     testSingleMethod(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol):
> Results should contain region
> test,bbb,1342509423473.2c0326188f899f3e91ec5eb623959c13. for row 'bbb'
>
> TestFromClientSide
>     testPoolBehavior(org.apache.hadoop.hbase.client.TestFromClientSide):
> expected:<3> but was:<4>
>
> TestZooKeeper
>     testClientSessionExpired(org.apache.hadoop.hbase.TestZooKeeper)
>
> TestReplication
>     testDisableInactivePeer(org.apache.hadoop.hbase.replication.TestReplication):
> Shutting down
>
> TestMasterReplication
>     testSimplePutDelete(org.apache.hadoop.hbase.replication.TestMasterReplication):
> Waited too much time for put replication
>
> TestMultiSlaveReplication
>     testMultiSlaveReplication(org.apache.hadoop.hbase.replication.TestMultiSlaveReplication):
> Unable to add peer
>
> TestReplicationPeer
>     testResetZooKeeperSession(org.apache.hadoop.hbase.replication.TestReplicationPeer):
> ReplicationPeer ZooKeeper session was not properly expired.
>
> I didn't get to them all before AWS yanked back my spot instances but
> I ordered the list from most likely to least, the remaining tests were
> in io.hfile.*, thrift.*, and util.*
>
> I'll circle back, confirm each individually, and open JIRAs with more detail.
>
> The cluster of replication test failures are a concern, but I've seen
> in other environments such as this one that the tests are timing
> dependent. On a slow or busy test system they can fail with "waited
> too much time ...". So a solution for this is to not use the system
> clock but instead EnvironmentEdge or whatever incremented only when
> the test process has CPU time. I haven't looked into this in detail
> yet.
>
> Best regards,
>
>    - Andy

On Tue, Jul 17, 2012 at 1:18 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> I'm on the hunt for flaky tests.
>

Thanks Andrew.  If a list, we can divvy up the fix work.

Good on you,
St.Ack

rat plugin OOM'ed again. (Or rather the JVM could not allocate memory via malloc). Sigh.

----- Original Message -----
From: Apache Jenkins Server <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc:
Sent: Monday, July 23, 2012 11:08 PM
Subject: Build failed in Jenkins: HBase-0.94 #359

See <https://builds.apache.org/job/HBase-0.94/359/changes>

Changes:

[larsh] HBASE-5659 Addendum

------------------------------------------
[...truncated 618 lines...]
Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.881 sec
Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.998 sec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.214 sec
Running org.apache.hadoop.hbase.util.TestPoolMap$TestRoundRobinPoolType
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec
Running org.apache.hadoop.hbase.util.TestPoolMap$TestThreadLocalPoolType
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec
Running org.apache.hadoop.hbase.util.TestPoolMap$TestReusablePoolType
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.05 sec
Running org.apache.hadoop.hbase.util.TestIdLock
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.342 sec
Running org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.197 sec
Running org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 175.642 sec
Running org.apache.hadoop.hbase.util.TestFSTableDescriptors
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.932 sec
Running org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.562 sec
Running org.apache.hadoop.hbase.util.TestMergeTool
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 303.727 sec
Running org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 406.341 sec
Running org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 359.122 sec
Running org.apache.hadoop.hbase.mapred.TestTableInputFormat
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 73.203 sec
Running org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.026 sec
Running org.apache.hadoop.hbase.util.TestRegionSplitter
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.827 sec
Running org.apache.hadoop.hbase.regionserver.TestStore
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.596 sec
Running org.apache.hadoop.hbase.util.TestHBaseFsck
Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 500.445 sec
Running org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.218 sec
Running org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.845 sec
Running org.apache.hadoop.hbase.regionserver.TestParallelPut
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.84 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.919 sec
Running org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.054 sec
Running org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.248 sec
Running org.apache.hadoop.hbase.regionserver.TestBlocksRead
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.245 sec
Running org.apache.hadoop.hbase.regionserver.TestMasterAddressManager
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.726 sec
Running org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.684 sec
Running org.apache.hadoop.hbase.regionserver.TestAtomicOperation
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.734 sec
Running org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.913 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.151 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLogBench
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.685 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.053 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.071 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLog
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.298 sec
Running org.apache.hadoop.hbase.regionserver.metrics.TestSchemaMetrics
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.099 sec
Running org.apache.hadoop.hbase.regionserver.TestRpcMetrics
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.021 sec
Running org.apache.hadoop.hbase.regionserver.TestStoreScanner
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.057 sec
Running org.apache.hadoop.hbase.regionserver.TestCacheOnWriteInSchema
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.614 sec
Running org.apache.hadoop.hbase.regionserver.TestSeekOptimizations
Tests

This was the first time I saw NumberFormatException in trunk build:

Running org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.667 sec
#
# A fatal error has been detected by the Java Runtime Environment:
#
NumberFormatException: #  SIGSEGV (0xb) at pc=0xf6c13079, pid=31814,
tid=4293966704
#
# JRE version: 6.0_27-b07
# Java VM: Java HotSpot(TM) Server VM (20.2-b06 mixed mode linux-x86 )
# Problematic frame:
NumberFormatException: # V  [libjvm.so+0x238079]  frame::frame(int*, int*)+0x19
#
# An error report file with more information is saved as:
# /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/hbase-server/hs_err_pid31814.log
#
NumberFormatException: # If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#
Aborted
org.apache.maven.surefire.booter.SurefireBooterForkException: Error
occurred in starting fork, check output in log
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:238)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$000(ForkStarter.java:64)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter$ParallelFork.call(ForkStarter.java:303)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter$ParallelFork.call(ForkStarter.java:285)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

 FYI

Rhe RAT plugin is OOM'ing, which generates hs_err_pidxxxx.log, which is then deemed to have an invalid license.
(see https://builds.apache.org/job/HBase-0.94-security/ws/trunk/target/rat.txt)

-- Lars

________________________________
From: Apache Jenkins Server <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Friday, July 13, 2012 6:25 PM
Subject: Build failed in Jenkins: HBase-0.94-security #41

See <https://builds.apache.org/job/HBase-0.94-security/41/changes>

Changes:

[larsh] CHANGES.txt/pom.xml for 0.94.1RC0 take two

[larsh] HBASE-6389 Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments (Aditya Kishore)

[stack] HBASE-6380 bulkload should update the store.storeSize

[jxiang] HBASE-6384 hbck should group together those sidelined regions need to be bulk loaded later

------------------------------------------
[...truncated 12807 lines...]
Setting project property: maven.dependency.org.apache.httpcomponents.httpclient.jar.path -> /home/jenkins/.m2/repository/org/apache/httpcomponents/httpclient/4.1.2/httpclient-4.1.2.jar
Setting project property: maven.dependency.org.apache.httpcomponents.httpcore.jar.path -> /home/jenkins/.m2/repository/org/apache/httpcomponents/httpcore/4.1.3/httpcore-4.1.3.jar
Setting project property: maven.dependency.org.jruby.jruby-complete.jar.path -> /home/jenkins/.m2/repository/org/jruby/jruby-complete/1.6.5/jruby-complete-1.6.5.jar
Setting project property: maven.dependency.org.mortbay.jetty.jetty.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar
Setting project property: maven.dependency.org.mortbay.jetty.jetty-util.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar
Setting project property: maven.dependency.org.mortbay.jetty.jsp-2.1.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar
Setting project property: maven.dependency.org.eclipse.jdt.core.jar.path -> /home/jenkins/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar
Setting project property: maven.dependency.org.mortbay.jetty.jsp-api-2.1.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar
Setting project property: maven.dependency.org.mortbay.jetty.servlet-api-2.5.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar
Setting project property: maven.dependency.org.codehaus.jackson.jackson-core-asl.jar.path -> /home/jenkins/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar
Setting project property: maven.dependency.org.codehaus.jackson.jackson-mapper-asl.jar.path -> /home/jenkins/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar
Setting project property: maven.dependency.org.codehaus.jackson.jackson-jaxrs.jar.path -> /home/jenkins/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.8/jackson-jaxrs-1.8.8.jar
Setting project property: maven.dependency.org.codehaus.jackson.jackson-xc.jar.path -> /home/jenkins/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.8/jackson-xc-1.8.8.jar
Setting project property: maven.dependency.org.slf4j.slf4j-api.jar.path -> /home/jenkins/.m2/repository/org/slf4j/slf4j-api/1.4.3/slf4j-api-1.4.3.jar
Setting project property: maven.dependency.org.slf4j.slf4j-log4j12.jar.path -> /home/jenkins/.m2/repository/org/slf4j/slf4j-log4j12/1.4.3/slf4j-log4j12-1.4.3.jar
Setting project property: maven.dependency.tomcat.jasper-compiler.jar.path -> /home/jenkins/.m2/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar
Setting project property: maven.dependency.tomcat.jasper-runtime.jar.path -> /home/jenkins/.m2/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar
Setting project property: maven.dependency.commons-el.commons-el.jar.path -> /home/jenkins/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar
Setting project property: maven.dependency.org.jamon.jamon-runtime.jar.path -> /home/jenkins/.m2/repository/org/jamon/jamon-runtime/2.3.1/jamon-runtime-2.3.1.jar
Setting project property: maven.dependency.com.google.protobuf.protobuf-java.jar.path -> /home/jenkins/.m2/repository/com/google/protobuf/protobuf-java/2.4.0a/protobuf-java-2.4.0a.jar
Setting project property: maven.dependency.com.sun.jersey.jersey-core.jar.path -> /home/jenkins/.m2/repository/com/sun/jersey/jersey-core/1.4/jersey-core-1.4.jar
Setting project property: maven.dependency.com.sun.jersey.jersey-json.jar.path -> /home/jenkins/.m2/repository/com/sun/jersey/jersey-json/1.4/jersey-json-1.4.jar
Setting project property: maven.dependency.org.codehaus.jettison.jettison.jar.path -> /home/jenkins/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar
Setting project property: maven.dependency.com.sun.xml.bind.jaxb-impl.jar.path -> /home/jenkins/.m2/repository/com/sun/xml/bind/jaxb-impl/2.1.12/jaxb-impl-2.1.12.jar
Setting project property: maven.dependency.com.sun.jersey.jersey-server.jar.path -> /home/jenkins/.m2/repository/com/sun/jersey/jersey-server/1.4/jersey-server-1.4.jar
Setting project property: maven.dependency.asm.asm.jar.path -> /home/jenkins/.m2/repository/asm/asm/3.1/asm-3.1.jar
Setting project property: maven.dependency.javax.xml.bind.jaxb-api.jar.path -> /home/jenkins/.m2/repository/javax/xml/bind/jaxb-api/2.1/jaxb-api-2.1.jar
Setting project property: maven.dependency.javax.activation.activation.jar.path -> /home/jenkins/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar
Setting project property: maven.dependency.stax.stax-api.jar.path -> /home/jenkins/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar
Setting project property: maven.dependency.junit.junit.jar.path -> /home/jenkins/.m2/repository/junit/junit/4.10-HBASE-1/junit-4.10-HBASE-1.jar
Setting project property: maven.dependency.org.mockito.mockito-all.jar.path -> /home/jenkins/.m2/repository/org

Interesting! I had the same error on my MacBookAir when I ran the full build. I assumed that we cannot have that many faults. This explains it.

On Jul 14, 2012, at 4:02 AM, lars hofhansl wrote:

> Rhe RAT plugin is OOM'ing, which generates hs_err_pidxxxx.log, which is then deemed to have an invalid license.
> (see https://builds.apache.org/job/HBase-0.94-security/ws/trunk/target/rat.txt)
>
> -- Lars
>
> ________________________________
> From: Apache Jenkins Server <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Friday, July 13, 2012 6:25 PM
> Subject: Build failed in Jenkins: HBase-0.94-security #41
>
> See <https://builds.apache.org/job/HBase-0.94-security/41/changes>
>
> Changes:
>
> [larsh] CHANGES.txt/pom.xml for 0.94.1RC0 take two
>
> [larsh] HBASE-6389 Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments (Aditya Kishore)
>
> [stack] HBASE-6380 bulkload should update the store.storeSize
>
> [jxiang] HBASE-6384 hbck should group together those sidelined regions need to be bulk loaded later
>
> ------------------------------------------
> [...truncated 12807 lines...]
> Setting project property: maven.dependency.org.apache.httpcomponents.httpclient.jar.path -> /home/jenkins/.m2/repository/org/apache/httpcomponents/httpclient/4.1.2/httpclient-4.1.2.jar
> Setting project property: maven.dependency.org.apache.httpcomponents.httpcore.jar.path -> /home/jenkins/.m2/repository/org/apache/httpcomponents/httpcore/4.1.3/httpcore-4.1.3.jar
> Setting project property: maven.dependency.org.jruby.jruby-complete.jar.path -> /home/jenkins/.m2/repository/org/jruby/jruby-complete/1.6.5/jruby-complete-1.6.5.jar
> Setting project property: maven.dependency.org.mortbay.jetty.jetty.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar
> Setting project property: maven.dependency.org.mortbay.jetty.jetty-util.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar
> Setting project property: maven.dependency.org.mortbay.jetty.jsp-2.1.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar
> Setting project property: maven.dependency.org.eclipse.jdt.core.jar.path -> /home/jenkins/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar
> Setting project property: maven.dependency.org.mortbay.jetty.jsp-api-2.1.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar
> Setting project property: maven.dependency.org.mortbay.jetty.servlet-api-2.5.jar.path -> /home/jenkins/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar
> Setting project property: maven.dependency.org.codehaus.jackson.jackson-core-asl.jar.path -> /home/jenkins/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar
> Setting project property: maven.dependency.org.codehaus.jackson.jackson-mapper-asl.jar.path -> /home/jenkins/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar
> Setting project property: maven.dependency.org.codehaus.jackson.jackson-jaxrs.jar.path -> /home/jenkins/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.8/jackson-jaxrs-1.8.8.jar
> Setting project property: maven.dependency.org.codehaus.jackson.jackson-xc.jar.path -> /home/jenkins/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.8/jackson-xc-1.8.8.jar
> Setting project property: maven.dependency.org.slf4j.slf4j-api.jar.path -> /home/jenkins/.m2/repository/org/slf4j/slf4j-api/1.4.3/slf4j-api-1.4.3.jar
> Setting project property: maven.dependency.org.slf4j.slf4j-log4j12.jar.path -> /home/jenkins/.m2/repository/org/slf4j/slf4j-log4j12/1.4.3/slf4j-log4j12-1.4.3.jar
> Setting project property: maven.dependency.tomcat.jasper-compiler.jar.path -> /home/jenkins/.m2/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar
> Setting project property: maven.dependency.tomcat.jasper-runtime.jar.path -> /home/jenkins/.m2/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar