Recently we had a discussion in our team regarding good testing practices (again) :)
With this post I try to summarize this discussion for myself.
The system under test
We were developing a piece of software exposing a REST-Interface using Spring for all kinds of things (REST-communication, DI, Hibernate integration, etc.)
Many of the tests already in place made use of the SpringTestRunner. Most of these tests were declared ‚unitests‘. I did not regard this as a good practice using test which boot up Spring (and quite a lot of the adjacent dependencies of the springbeans) as our lowest level of tests. Some team members defended those tests as it is the most convenient way for them to write tests. Anything less was not regarded helpful to show any meaningful failures.
In my book a unit test does not need the presence of a containing framework. The test take less time and you can express your dependencies more clearly when you state them explicitly. It helps me to mock out those dependencies if I don’t need them in my test.
Spring should only help to glue those pieces seamlessly together.
But we thought this may not be a Spring issue, so we tried to take this discussion to a more general level.
Taking it to a general level
Basically it boiled down to the questions:
- How do we test?
- In which extend do we test each layer?
- Of which granularity should our tests be?
We agreed to some extent to applying TDD and
having read [Growing Object-Oriented Software, Guided by Tests]1, we assumed we had a similar understanding of what ‚testing‘ means regarding that context.
To quickly summarize [Growing Object-Oriented Software, Guided by Tests]1:
The book describes two essential feedback loops:
- One outer loop, containing the acceptance tests
- Several inner loops containing the unit_test
Comparing our way we drive development by tests, we encountered subtle differences
- I put more emphasis on the unit test
- My teammate put more effort in acceptance tests or coarse grained tests, as the seem to have more realworld value
To understand where this difference might come from, we got back to the picture (pictures do always help, don’t they ;) of the TestPyramid:
- In my case it took the form of the pyramid, putting most emphasis on the unit layer.
- In his case it took the form of a diamond, having more enphasis in the middle layer.
As one result my teammate stated, he was not fond of the term ‚unittest‘ as it is too vague for him. In his eyes it could mean anything. It may even be a strictly technical term derivied from the TestRunner, as a testcase is often implemented as a JUnit Test (yes, we have a Java background). So every test done by JUnit could possibly be regarded as a Unit test.
I disagreed and insisted that the term Unit in Object-oriented software is commonly defined as of being the class you’re looking at.
He proposed a different term for that matter.
For above reasons i rejected this for now as I saw no need for another overloading of words.
As we didn’t come to a conclusion, we agreed on doing do a short survey in our team, and possibly elsewhere, stating the question:
‚How do you define the term unit in Object Oriented Software?‘
As a result we had (surprisingly) many definitions of „unit“
- Everything run by JUnit
- Unit is a class
- Unit is everything
- Unit is not defined
I took this discussion outside of our team and asked the same question. While some quickly replied with ‚A unit refers to a class in oo‘, I got one rather interesting definition:
Everything that has an interface I can write tests
against. Have hit the occasional wall with this
If I can test it in isolation (i.e., without db, etc.) I
consider it a unit. Anything else is blurry.
While I question part one of this statement. (A UI has an interface but tests using this interface are mostly no unit tests), part two is rather important to me.
A unit test with all its dependencies mocked out is clearly tested in isolation23. It depends on the thing and the intent of your test, how much you loosen this isolation statement. You may not need to mock out your domainmodel, perhaps you don’t need to mock out some other helper structures, if they don’t blur the intent of your test. But things like a DB, a service call, a complex algorithm or be it some framework dependency (JSF or maybe Spring), might violate the isolation of your test to some degree and possible test failures might be harder to spot.
Coming back to the starting point
I still don’t think that the SpringRunnableTest serve a good purpose as a unittest, when you apply a TDD approach. They come with a additional load and for many testcases you don’t need Spring in place, though it might it might come handy at times that your dependencies are kicked in externally. I still do like them, when used in higher layers of the testpyramid.
If the part you are about to build is not that complex – say some more straightforward CRUD part – SpringRunnableTest (as a more diamond like approach) might serve you well.
Also if you do prefer a more ‚acceptance-test-centric‘ approach (with Spring in place and the testconfiguration at your service), you might treat and design them in a unittest-like fashion.
But I’ve often seen those tests degenerate from a unittest to an integration test (yet still be named unittest), becoming dependent on the outside world and becoming slower and more brittle.
Another colleague showed us a handy technique using Mockito that I haven’t come across yet (Though I used mockito before, shame on me!). To avoid Spring dependency injection, and to avoid handrolling all your mocks Mocktio comes with a similar feature, that allows you to define mocks and inject them in a more concise manner link.
I kind of like this approach, but some of my initial criticism kicks in here as well. The definition of your dependencies becomes more implicit. But compared to starting up Spring this approach seems to be way faster and your tests become more readable.
What about isolation? As I see it, my CuT is still run as isolated as done without mockito-trickery, only your test-class becomes dependent on mockito.
This may be the best of both worlds, let’s see if this technique helps that a unittest remains a unittest over time.
- http://www.amazon.de/Growing-Object-Oriented-Software-Guided-Signature/dp/0321503627 „Growing Object-Oriented Software, Guided by Tests“ ↩ ↩
- http://martinfowler.com/articles/mocksArentStubs.html#TestIsolation „Martin Fowler, Mocks aren’t Stubs“ ↩
- http://artofunittesting.com/definition-of-a-unit-test/ „The Art of Unittesting“ ↩