Most forms of testing are about isolating a particular characteristic of your system, and exploring it from a risk perspective. It could be:
Given this, the best approach to test planning should be risk-based: consider which risks you want to mitigate, and test accordingly:
This should work at every level within a project. If you are building a new feature, you should consider:
Often, the decision of whether to automate a test will be based on whether or not it can be expressed objectively. For example, checking that a REST endpoint “returns the right error code” is objective, and is therefore a candidate for automation.
Automated tests look roughly the same, irrespective of the scope they are trying to test.
A useful way to think about automated testing is that it turns the System Under Test into a Pure Function: This means that for a specific set of inputs, the system will produce a specific output, reliably, every time.
Getting complex systems to behave as pure functions can be costly, but there are techniques to help with this such as Mocking. However, if you try to devise as much of your software in a pure-functional way to start with, automated testing is much easier.
Automated Testing has an interesting effect on managing Complexity Risk: Although you may initially write a Unit Test (say) to mitigate the risk of having implemented a feature wrongly, you are also given insurance against future change breaking that feature. That is to say, they are regression tests. However, implementing tests like this is better than building regression tests, as discussed here.
– how do automated tests mitigate complexity risk? tbd
Manual Testing is, at some level, essential if your product is to be used by humans. Although UI-Automation tools such as Selenium allow you to script browser interactions, they cannot reliably catch every problem.
For example, ensuring the UI “looks ok and doesn’t glitch” is entirely subjective: you’ll need to express this in a manual test. Manual Tests are often described in Test Plans and Test Scripts in order to ensure repeatability, and manage Process Risk.
Since manual tests carry much higher per-use cost to run, there is a tendency to want to save this cost by doing fewer releases. After all, fewer releases means less manual testing, but this may increase Process Risk.
How do you decide whether to keep a test manual, or automate? The more automated a test is, the more cheaply it can be re-used. However, the process of automation can take longer, and so adds Schedule Risk. Whether or not it’s worth automating is to some extend going to depend on how much you value future time.
In the initial conception, Black-Box Testing ignores the implementation details of a component and tests the interface only.
White-box testing however considers the components within the box, and how they interact with one another in order to define the tests. This is fair enough if, for some reason, you are unable to test the components individually for some reason: knowing how something is implemented gives you an insight into where the bugs will hide, and therefore, where the risks lie.
However, if possible, it’s better to break open the white box and test the components themselves. This means you end up having “higher” and “lower” level tests, depending on the scope of the System Under Test. There are several advantages to this:
Expanding on this then, the Testing Pyramid idea is that lower level, automated tests which run quickly should be common, while there should be fewer of the more expensive “whole system” level tests.
Finally, since manual tests are run by people (who are comparatively slow and costly), these should be the rarest kind of test.
Sometimes, testing is handled by external teams (possibly in other locales). This is often done as a cost-saving measure, but comes with some penalties such as:
Also called test-first development, the idea here (from Extreme Programming) is that you write the tests before the code, in order that you think up-front about the requirements of the software you are writing. The aim of this is to minimize Complexity Risk via preventing developers from Gold Plating, and getting them to do The Simplest Thing That Can Possibly Work.
Additionally, by having test fail before they pass, you mitigate the risk of writing a “null” test (see below).
Code Coverage tools are a useful way of showing you which parts of your software might contain bugs due to lack of testing, which is really useful in the Risk Evaluation phase of test-planning.
Sometimes code coverage spawns its own Map And Territory Risks though, where people forget that the goal should be mitigating overall project risk (via delivering functionality and so forth) and start to believe that the goal is delivering 100% code coverage. Writing tests to cover every
get() method is a fools’ errand which increases the overall codebase complexity for no real reduction in Feature Risk.
Worse still is that having 100% code coverage does not guarantee an absence of bugs, or that the code will do what the users wanted it to do. Feature Risk is always there.
There are so many different types of testing and this guide is not meant to be exhaustive. Instead, here is a table covering some of the main types of testing and the risks they mitigate:
|Boundary Risk||System Integration Testing
User Acceptance Testing
|Dependency Risk||Integration Testing
System Integration Testing
|Production Risk||Performance Testing / Load Testing
Disaster Recovery Testing
Smoke / Sanity Testing
|Software Risk||Unit Testing
|Feature Risk||Browser-Based Testing
Acceptance Testing (UAT)
|Visibility Risk||Usability Testing
|Complexity Risk||Unit Testing
Automated Acceptance testing
Firstly, it can be easy to fool yourself with tests: just because your tests pass does not mean your code is perfect. Vigilance is required against Map And Territory Risk:
Second, Testing is a double-edged sword. While it allows you to mitigate various Feature Risks, by adding test-code to your project you are necessarily increasing the complexity. Maintaining tests is hard work, and if you’re not careful, running tests can take time and slow down builds and add delay through Process Risk.
Third, if you are exploring functionality in order to flush out requirements, understand user behaviour or figure out performance characteristics, then there is no point in building tests yet: what you are doing is exploratory at best and the extra code will slow you down.
For these reasons, focus on writing the smallest number of tests that mitigates the risks.
Risk Based Agile Testing by Martin Ivison, which covers a lot of this ground in much more detail.
How does this help?
Found this interesting? Please add your star on GitHub to be invited to join the Risk-First GitHub group.