Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - random failure on tests


Copy link to this message
-
Re: random failure on tests
Nicolas Liochon 2013-05-29, 07:12
Hello,

Option 1:
We still have some flaky tests. You can benchmark you build against
https://builds.apache.org/job/HBase-TRUNK/ and
https://builds.apache.org/job/hbase-0.95/
You can also use this tool: https://github.com/jeffreyz88/jenkins-tools to
get a review on the last fails:

On 0.95, we have some tests that failed ~15% of the time:

org.apache.hadoop.hbase.client.testadmin.testforcesplitmultifamily
80/6/13     1    1    1    1    1    1    1    1    1    1    1    1
-1    0   -1
org.apache.hadoop.hbase.client.testhcm.testdeleteforzkconnleak 86/6/6
1   -1    0    1    1    1    1    1    1    1    1    1    1    1    1
org.apache.hadoop.hbase.client.testmultiparallel.testactivethreadscount
86/6/6     1    1    1    1    1    1    1   -1    0    1    1    1    1
1    1
org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort
86/6/6     1    1    1    1    1    1    1   -1    0    1    1    1    1
1    1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
86/6/6     1    1    1    1    1    1    1    1    1    1   -1    0    1
1    1
org.apache.hadoop.hbase.rest.client.testremoteadmin.testclusterstatus
86/6/6     1    1    1    1    1    1    1    1    1    1    1    1   -1
0    1
org.apache.hadoop.hbase.security.access.testaccesscontroller.testglobalauthorizationfornewregisteredrs
86/6/6     1    1    1    1    1    1    1   -1    0    1    1    1    1
1    1
org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta
86/6/6
Option 2:
You've got some issues in you env. We run the test in parallel (5 b default
when you run all tests, 2 when you run only the small & medium ones). 5
requires around 10 GB or memory. If you have less or if the built is
shared, you may enter into strange conditions around test timing
requirements.
You also need to use oracle jdk.

See as well http://hbase.apache.org/book.html / 15.7.3. Running tests, for
extra parameters.
These options are not exclusive. It seems that these days trunk build is ok
~80% of the time these days, and 0.95 50%. You should expect something
similar.
A test becomes flaky because a patch breaks it just a little. The patch
passes the peer review and the precommit runs, but after a while the
randomness shows up, and we need to fix the code again. It's a never ending
story. Any help in fixing them is always greatly appreciated.

Cheers,

Nicolas
On Wed, May 29, 2013 at 8:21 AM, Lisen Mu <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I'm setting up a jenkins job for hbase, building branch 0.95 (from github)
> under jdk 6.
>
> However sometimes the build pasts, sometimes does not. Several tests are
> likely to fail, such as:
>
>
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testTooManyDelayedRpcs
> org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak
>
> yet they do not fail every time.
>
> Any clue about what might be the problem? Thanks.
>
>
> From the last failed build:
>
> the executed mvn command line:
>
> Executing Maven:  -B -f
> /var/lib/jenkins/jobs/HBase-0.95-jdk-6/workspace/pom.xml clean package
>
>
>
> Possibly related log:
>
>
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue
>
> Error Message
>
> Index: 1, Size: 1
>
> Stacktrace
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.get(ArrayList.java:322)
>         at
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpc(TestDelayedRpc.java:112)
>         at
> org.apache.hadoop.hbase.ipc.TestDelayedRpc.testDelayedRpcImmediateReturnValue(TestDelayedRpc.java:71)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)