Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Disable flaky tests, and let Jenkin stay blue?


Copy link to this message
-
Re: Disable flaky tests, and let Jenkin stay blue?
I'm between +0 and -0.5
+0 because I like green status. They help to detect regression.

-0.5 because
   - If we can't afford to fix it now I guess it won't be fixed in the
future:  we will continue to keep it in the codebase (i.e. paying the cost
of updating it when we change an interface), but without any added value as
we don't run it.
   - some tests failures are actually issues in the main source code. Ok,
they're often minor, but still they are issues. Last example I have is from
today: the one found by Jeff related HBASE-8204.
    - and sometimes it shows lacks in the way we test (for example, the
waitFor stuff, while quite obvious in a way, was added only very recently).
    - often a flaky test is better than no test at all: they can still
detect regressions.
    - I also don't understand why the precommit seems to be now better than
the main build.

For me, doing it in a case by case way would be simpler (using the
component owners: it a test on a given component is flaky, the decision can
be taken between the people who want to remove the test and the component
owners, with a jira, an analysis and a traced decision)

Cheers,

Nicolas

On Tue, Apr 2, 2013 at 7:09 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:

> We have not seen couple blue Jenkin builds for 0.95/trunk for quite some
> time.  Because of this, sometimes we ignore the precommit build failures,
> which could let some bugs (code or test) sneaked in.
>
> I was wondering if it is time to disable all flaky tests and let Jenkin
> stay blue.  We can maintain a list of tests disabled, and get them back
> once they are fixed. For each disabled test, if someone wants to get it
> back, please file a jira so that we don't duplicate the effort and work on
> the same one.
>
> As to how to define a test flaky, to me, if a test fails twice in the last
> 10/20 runs, then it is flaky, if there is no apparent env issue.
>
> We have different Jenkins job for hadoop 1 and hadoop 2.  If a test is
> flaky for either one, it is flaky.
>
> What do you think?
>
> Thanks,
> Jimmy
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB