I'm generally on the same page here, but this is what I'm (currently)
thinking about how it could work:
We need to characterize the unit tests because some are longer than
others, and we want to split them up in order of complexity.

Keywal, this next bit is divergent from what I was saying the other day...

The next level up should be testing between integration of components.
Again this should be run by the CI; really this is more of a finer
grain separation of what was previously just unit tests.
I've revised my thinking such that this could a different category of
unit tests (maybe @Integration that run in their own jvm, in a
different step).
This helps separate concerns between the individual class testing and
the testing of multiple pieces working together.

Onto using the failsafe plugin.

I like the idea of really leveraging the failsafe plugin to spin up
the tests on some of 'real' instance. The (canonical) simple example
of using failsafe is to spin up a Jetty server and test whatever stuff
you are doing with a 'real'

Also, integrating with BigTop to do some of that stuff would also be
pretty sweet (and good for both communities).

A good place for these tests would be the hbase-it module discussed in
HBASE-4336, since that module can just depend on the rest of the
modules (giving full access to the components). Within that package,
we could conceivably have the multi-item integration tests
(@Integration form above) and then the 'real cluster' tests as well.

+1 Having tests that just use the top level API (e.g. what the FB guys
are using on their dev cluster after running the full test suite) is
really important to make sure we 'real' test cases to ensure
functionality.

Also, agreeing with Roman, making the plugin to enable that is going
to rough, but we should be able to at least find a way to (at least
manually at first) have those api level tests run on a real cluster,
even if it is a company X and they just post the results. Even then a
weekly build/run of that suite could be sufficient at first (though
having that for every commit would be must better - anyone know
someone at Amazon we can scrounge credits from?).

However, I do think that if tests start failing in _any_ of the above
phases then the patch that caused it needs to not be committed OR the
tests need to be revised (which would also be part of the patch, so my
original point stands). That is indicative of broken functionality and
trunk should be kept as stable as possible.

Yeah, devs shouldn't have to run the full set of tests before
submitting a patch, but they should easily be able to run everything
up to (but not including) the api-level testing.
What would be cool though would be that they could even run the
api-level testing, but that is just runs on a local MiniCluster, but
that's a lot of work out :).
A lot of that stability comes form having a staged testing cycle as
well as helping devs cut down their testing cycles, so anything we do
to help cut that time down is good stuff in my book..

-- Jesse
-------------------

ps. I hate to be 'that guy' but we need a consistent way of describing
the tests we talking about, but lets push that off until figure out
what the heck we actually want to do.

Jesse Yates
240-888-2200
@jesse_yates

> ---------- Forwarded message ----------
> From: Ted Yu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Date: Mon, 28 Nov 2011 12:23:43 -0800
> Subject: scoping integration tests
> The following discussion is closely related to HBASE-4712.
>
> We should reach general consensus so that the execution of future test
> strategy is smooth.
>
> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <[EMAIL PROTECTED]
> >wrote:
>
> > I was considering 'integration tests' as a separate concern from the
> > large/medium/small _unit_ tests.
> >
> > That is, in fact, why the failsafe plugin was added (and is designed
> for).
> >
> > Currently, we have a lot of tests that fall in the realm of integration
> > tests (testing integration between various pieces, rather than single
Jesse Yates
240-888-2200
@jesse_yates

On Thu, Dec 1, 2011 at 6:34 PM, Jesse Yates <[EMAIL PROTECTED]> wrote:
> We need to characterize the unit tests because some are longer than
> others, and we want to split them up in order of complexity.
>

Nkeywal has done this, no?  He's made them into small/medium/large.
Some tests may need to be moved between categories but the hard graft
has been done here?

You introduced the IntegrationTest category that is run with failsafe.
 Some of the current large unit tests might get moved over to this new
IntegrationTest category.
> Keywal, this next bit is divergent from what I was saying the other day...
>
> The next level up should be testing between integration of components.

I'm not sure I grok this.  Next level up from unit tests?   But unit
tests, for both medium and large, are allowed spin up mini clusters so
they are already testing integration of components?  If you are
suggesting that these kind of minicluster tests get moved out to
IntegrationTest category, then I'd disagree; or, I disagree that
should happen any time soon (too much work, too disruptive).

> Again this should be run by the CI;

If you are saying the IntegrationTests should be run the CI, I'm
thinking not.  I'm thinking they'd be run on occasion when trying to
proof a build or release candidate or trying to verify an hotfix
before production deploy

> I've revised my thinking such that this could a different category of
> unit tests (maybe @Integration that run in their own jvm, in a
> different step).
> This helps separate concerns between the individual class testing and
> the testing of multiple pieces working together.
>

I don't think this viable.

What I do think viable is committing the last nkeywal suggestion of
'mvn test' running current small and medium tests but then we have the
CI run small/medium/large.  Then, IntegrationTests are a new category
altogether of tests that go against the API only and that presume a
cluster (built by failsafe or otherwise); the tests do not do cluster
setup/teardown.

> A good place for these tests would be the hbase-it module discussed in
> HBASE-4336, since that module can just depend on the rest of the
> modules (giving full access to the components). Within that package,
> we could conceivably have the multi-item integration tests
> (@Integration form above) and then the 'real cluster' tests as well.
>

I'd be down w/ this.  Currently IntegrationTest* are mixed in w/ unit
tests but it would be best if they were off in a module of their own.
> +1 Having tests that just use the top level API (e.g. what the FB guys
> are using on their dev cluster after running the full test suite) is
> really important to make sure we 'real' test cases to ensure
> functionality.
>
> Also, agreeing with Roman, making the plugin to enable that is going
> to rough, but we should be able to at least find a way to (at least
> manually at first) have those api level tests run on a real cluster,
> even if it is a company X and they just post the results. Even then a
> weekly build/run of that suite could be sufficient at first (though
> having that for every commit would be must better - anyone know
> someone at Amazon we can scrounge credits from?).
>

+1 on above too.

> However, I do think that if tests start failing in _any_ of the above
> phases then the patch that caused it needs to not be committed OR the
> tests need to be revised (which would also be part of the patch, so my
> original point stands). That is indicative of broken functionality and
> trunk should be kept as stable as possible.
>

Depends.  Patch may have been committed before an IntegationTest is
run.  I think we should allow this.  For sure, if IntegrationTests are
broke, we need to figure which patch broke things.

> Yeah, devs shouldn't have to run the full set of tests before
> submitting a patch, but they should easily be able to run everything
> up to (but not including) the api-level testing.

Yes.  Nkeywal has this going on in his patch.

I'd say too that the failsafe plugin should out of the box let devs
run all IntegrationTest.  failsafe default might be to spin up a mini
hbase cluster.
I don't think so.  In fact I think it a prereq for easy IntegrationTest dev'ing.
The way I see it, I think we have a proposal on what to do.  Nkeywal
has done a refactoring of our unit tests into categories but we need a
new category of tests, tests that are optionally long-running against
clusters of variable size doing scales beyond what would be tenable on
Apache's or anyone's CI.  These are the tests we'd give our bigtop
brothers and that we'd run when we wanted to do a verification beyond
what our unit tests can do.

Good on you Jesse,
St.Ack

I may be splitting hairs while beating a dead horse, but just to get
closure on the issue, my last (probably) 2 cents, in line.
On Thu, Dec 1, 2011 at 10:30 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Thu, Dec 1, 2011 at 6:34 PM, Jesse Yates <[EMAIL PROTECTED]>
> wrote:
> > We need to characterize the unit tests because some are longer than
> > others, and we want to split them up in order of complexity.
> >
>
> Nkeywal has done this, no?  He's made them into small/medium/large.
> Some tests may need to be moved between categories but the hard graft
> has been done here?
>
> You introduced the IntegrationTest category that is run with failsafe.
>  Some of the current large unit tests might get moved over to this new
> IntegrationTest category.
>
>
I feel like most of those would actually go in the new @IntegrationTest
category.

You don't mean that they would be run by the failsafe plugin though, right?
I'm fine with making sure they are more frequently run than the api-level
tests.
>
> > Keywal, this next bit is divergent from what I was saying the other
> day...
> >
> > The next level up should be testing between integration of components.
>
> I'm not sure I grok this.  Next level up from unit tests?
Next level up in complexity was my intention here. Medium and Large kinda
cover this.
> But unit
> tests, for both medium and large, are allowed spin up mini clusters so
> they are already testing integration of components?  If you are
> suggesting that these kind of minicluster tests get moved out to
> IntegrationTest category, then I'd disagree; or, I disagree that
> should happen any time soon (too much work, too disruptive).
>

> > Again this should be run by the CI;
>
> If you are saying the IntegrationTests should be run the CI, I'm
> thinking not.  I'm thinking they'd be run on occasion when trying to
> proof a build or release candidate or trying to verify an hotfix
> before production deploy
>

This is why I think semantics are good groundwork :)

I would argue that unit tests really do test the individual unit (class)
and then integration tests each multiple components. The reason I make the
differentiation is that often times to test a single component properly a
bigger test (possibly on a mini cluster) is needed, but it is not
necessarily testing multiple pieces.

Based on you definition of Integration Test, then no, CI would not be the
way to go. However, based on my above with the @Integration annonation,
then they are not big enough to be a real problem.

Howver, they do kind of work as the 'large' category too. Maybe that should
just be renamed? This is not a sticking point for me - though I have been
harping on it - if the word 'integration' is becoming overloaded.

I'm okay with the current labels, the labeling sematics just feel a little
heavy handed.

+/- 0

>
> > I've revised my thinking such that this could a different category of
> > unit tests (maybe @Integration that run in their own jvm, in a
> > different step).
> > This helps separate concerns between the individual class testing and
> > the testing of multiple pieces working together.
> >
>
> I don't think this viable.
>
> What I do think viable is committing the last nkeywal suggestion of
> 'mvn test' running current small and medium tests but then we have the
> CI run small/medium/large.  Then, IntegrationTests are a new category
> altogether of tests that go against the API only and that presume a
> cluster (built by failsafe or otherwise); the tests do not do cluster
> setup/teardown.
>
I worry about code breaking those situations and we only find out when
running the bigger tests when we cut the releases or choose to do
verification. Unless that choice comes weekly, its is going to be a much
bigger pain trying to figure out which patch caused breakage.

Again, a semantic issue where I would have most of the tests that spin up
miniclusters as integration tests not the api tests.
> > Yeah, devs shouldn't have to run the full set of tests before
+1 on that being the default and then having a profile that bigtop uses to
deploy out to a real cluster. Given that the default would be to spin the
minicluster, it would be sweet if CI could run those too. Its another level
of assurance (it runs on the mini cluster so I'm _pretty sure_ it works)
and one that I would be super comfortable pushing things into the codebase
with (without it feels a little risky).

Maybe we could have a nightly/weekly jenkins build that just pulls the
current codebase and runs those, rather than for every commit?

That would keep the load on the system as low as possible and at the same
time help us minimize the time to find the tests.

So either way, as long as we have a fairly easy way to run them, it should
alright :)