Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> random failure on tests


Copy link to this message
-
Re: random failure on tests
Yes for the stack size. See as well the config 2.1.2.5, nofile & nproc are
important for the tests.
Check as well with the resource checker (see chapter 15.7.3.7) that the
values actually used in the tests execution are the ones that you expect.
You want MaxFileDescriptor to be above 2K to be safe. We run the tests with
a value set to 65K. Same for the nproc, in linux it actually controls the
number of java thread you can create.
On Wed, May 29, 2013 at 12:02 PM, Lisen Mu <[EMAIL PROTECTED]> wrote:

> Nicolas,
>
> Thanks for the reply!
>
> I run small & medium tests only, and I have 8G ram on the build server. My
> passing rate is also around 50%, but I'm not running -PrunAllTests.
>
> I can find 2 differences between official build and mine: missing prebuild
> step 'rm -rf /tmp/hbase-jenkins/hbase' and stack size is 1024 vs 8192 in
> ulimit. Could this be a problem? I would try out anyway. Thanks!
>
>
>
>
>
> On Wed, May 29, 2013 at 3:12 PM, Nicolas Liochon <[EMAIL PROTECTED]>
> wrote:
>
> > Hello,
> >
> > Option 1:
> > We still have some flaky tests. You can benchmark you build against
> > https://builds.apache.org/job/HBase-TRUNK/ and
> > https://builds.apache.org/job/hbase-0.95/
> > You can also use this tool: https://github.com/jeffreyz88/jenkins-toolsto
> > get a review on the last fails:
> >
> > On 0.95, we have some tests that failed ~15% of the time:
> >
> > org.apache.hadoop.hbase.client.testadmin.testforcesplitmultifamily
> > 80/6/13     1    1    1    1    1    1    1    1    1    1    1    1
> > -1    0   -1
> > org.apache.hadoop.hbase.client.testhcm.testdeleteforzkconnleak 86/6/6
> > 1   -1    0    1    1    1    1    1    1    1    1    1    1    1    1
> > org.apache.hadoop.hbase.client.testmultiparallel.testactivethreadscount
> > 86/6/6     1    1    1    1    1    1    1   -1    0    1    1    1    1
> > 1    1
> >
> org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort
> > 86/6/6     1    1    1    1    1    1    1   -1    0    1    1    1    1
> > 1    1
> >
> >
> org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
> > 86/6/6     1    1    1    1    1    1    1    1    1    1   -1    0    1
> > 1    1
> > org.apache.hadoop.hbase.rest.client.testremoteadmin.testclusterstatus
> > 86/6/6     1    1    1    1    1    1    1    1    1    1    1    1   -1
> > 0    1
> >
> >
> org.apache.hadoop.hbase.security.access.testaccesscontroller.testglobalauthorizationfornewregisteredrs
> > 86/6/6     1    1    1    1    1    1    1   -1    0    1    1    1    1
> > 1    1
> > org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta
> > 86/6/6
> >
> >
> > Option 2:
> > You've got some issues in you env. We run the test in parallel (5 b
> default
> > when you run all tests, 2 when you run only the small & medium ones). 5
> > requires around 10 GB or memory. If you have less or if the built is
> > shared, you may enter into strange conditions around test timing
> > requirements.
> > You also need to use oracle jdk.
> >
> > See as well http://hbase.apache.org/book.html / 15.7.3. Running tests,
> for
> > extra parameters.
> >
> >
> > These options are not exclusive. It seems that these days trunk build is
> ok
> > ~80% of the time these days, and 0.95 50%. You should expect something
> > similar.
> >
> >
> > A test becomes flaky because a patch breaks it just a little. The patch
> > passes the peer review and the precommit runs, but after a while the
> > randomness shows up, and we need to fix the code again. It's a never
> ending
> > story. Any help in fixing them is always greatly appreciated.
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> > On Wed, May 29, 2013 at 8:21 AM, Lisen Mu <[EMAIL PROTECTED]> wrote:
> >
> > > Hello,
> > >
> > > I'm setting up a jenkins job for hbase, building branch 0.95 (from
> > github)
> > > under jdk 6.
> > >
> > > However sometimes the build pasts, sometimes does not. Several tests
> are
> > > likely to fail, such as:
> > >
> > >
> > >
> >
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB