I may be splitting hairs while beating a dead horse, but just to get
closure on the issue, my last (probably) 2 cents, in line.
On Thu, Dec 1, 2011 at 10:30 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Thu, Dec 1, 2011 at 6:34 PM, Jesse Yates <[EMAIL PROTECTED]>
> > We need to characterize the unit tests because some are longer than
> > others, and we want to split them up in order of complexity.
> Nkeywal has done this, no? He's made them into small/medium/large.
> Some tests may need to be moved between categories but the hard graft
> has been done here?
> You introduced the IntegrationTest category that is run with failsafe.
> Some of the current large unit tests might get moved over to this new
> IntegrationTest category.
I feel like most of those would actually go in the new @IntegrationTest
You don't mean that they would be run by the failsafe plugin though, right?
I'm fine with making sure they are more frequently run than the api-level
> > Keywal, this next bit is divergent from what I was saying the other
> > The next level up should be testing between integration of components.
> I'm not sure I grok this. Next level up from unit tests?
Next level up in complexity was my intention here. Medium and Large kinda
> But unit
> tests, for both medium and large, are allowed spin up mini clusters so
> they are already testing integration of components? If you are
> suggesting that these kind of minicluster tests get moved out to
> IntegrationTest category, then I'd disagree; or, I disagree that
> should happen any time soon (too much work, too disruptive).
> > Again this should be run by the CI;
> If you are saying the IntegrationTests should be run the CI, I'm
> thinking not. I'm thinking they'd be run on occasion when trying to
> proof a build or release candidate or trying to verify an hotfix
> before production deploy
This is why I think semantics are good groundwork :)
I would argue that unit tests really do test the individual unit (class)
and then integration tests each multiple components. The reason I make the
differentiation is that often times to test a single component properly a
bigger test (possibly on a mini cluster) is needed, but it is not
necessarily testing multiple pieces.
Based on you definition of Integration Test, then no, CI would not be the
way to go. However, based on my above with the @Integration annonation,
then they are not big enough to be a real problem.
Howver, they do kind of work as the 'large' category too. Maybe that should
just be renamed? This is not a sticking point for me - though I have been
harping on it - if the word 'integration' is becoming overloaded.
I'm okay with the current labels, the labeling sematics just feel a little
> > I've revised my thinking such that this could a different category of
> > unit tests (maybe @Integration that run in their own jvm, in a
> > different step).
> > This helps separate concerns between the individual class testing and
> > the testing of multiple pieces working together.
> I don't think this viable.
> What I do think viable is committing the last nkeywal suggestion of
> 'mvn test' running current small and medium tests but then we have the
> CI run small/medium/large. Then, IntegrationTests are a new category
> altogether of tests that go against the API only and that presume a
> cluster (built by failsafe or otherwise); the tests do not do cluster
I worry about code breaking those situations and we only find out when
running the bigger tests when we cut the releases or choose to do
verification. Unless that choice comes weekly, its is going to be a much
bigger pain trying to figure out which patch caused breakage.
Again, a semantic issue where I would have most of the tests that spin up
miniclusters as integration tests not the api tests.
> > Yeah, devs shouldn't have to run the full set of tests before
+1 on that being the default and then having a profile that bigtop uses to
deploy out to a real cluster. Given that the default would be to spin the
minicluster, it would be sweet if CI could run those too. Its another level
of assurance (it runs on the mini cluster so I'm _pretty sure_ it works)
and one that I would be super comfortable pushing things into the codebase
with (without it feels a little risky).
Maybe we could have a nightly/weekly jenkins build that just pulls the
current codebase and runs those, rather than for every commit?
That would keep the load on the system as low as possible and at the same
time help us minimize the time to find the tests.
So either way, as long as we have a fairly easy way to run them, it should