Fork me on GitHub
Risk First Logo

Home of Risk-First Software Development



Start Here


Home
Contributing
Quick Summary
A Simple Scenario
The Risk Landscape

Discuss


Please star this project in GitHub to be invited to join the Risk First Organisation.

Blog


Consuming Hacker News 14 May 2019
Twitter Cards 22 March 2019
Next - Scrum? 7 March 2019
Available On Amazon 25 February 2019
New Proof 10 February 2019

Publications



Click Here For Details


Under Construction

What Is It

Most forms of testing are about isolating a particular characteristic of your system, and exploring it from a risk perspective. It could be:

How It Works

Testing Process

The whole purpose of testing is to meet reality early, ahead of putting software in front of real users, where you face Production Risks, like reputation damage and financial penalties.

Given this, the best approach to test planning should be risk-based: consider which risks you want to mitigate, and test accordingly:

Examples

This should work at every level within a project. If you are building a new feature, you should consider:

Where It’s Used

Variations

Automated Tests

Often, the decision of whether to automate a test will be based on whether or not it can be expressed objectively. For example, checking that a REST endpoint “returns the right error code” is objective, and is therefore a candidate for automation.

Automated tests look roughly the same, irrespective of the scope they are trying to test.

Testing Process

A useful way to think about automated testing is that it turns the System Under Test into a Pure Function: This means that for a specific set of inputs, the system will produce a specific output, reliably, every time.

Getting complex systems to behave as pure functions can be costly, but there are techniques to help with this such as Mocking. However, if you try to devise as much of your software in a pure-functional way to start with, automated testing is much easier.

Automated Testing has an interesting effect on managing Complexity Risk: Although you may initially write a Unit Test (say) to mitigate the risk of having implemented a feature wrongly, you are also given insurance against future change breaking that feature. That is to say, they are regression tests. However, implementing tests like this is better than building regression tests, as discussed here.

– how do automated tests mitigate complexity risk? tbd

Manual Tests

Manual Testing is, at some level, essential if your product is to be used by humans. Although UI-Automation tools such as Selenium allow you to script browser interactions, they cannot reliably catch every problem.

For example, ensuring the UI “looks ok and doesn’t glitch” is entirely subjective: you’ll need to express this in a manual test. Manual Tests are often described in Test Plans and Test Scripts in order to ensure repeatability, and manage Process Risk.

Since manual tests carry much higher per-use cost to run, there is a tendency to want to save this cost by doing fewer releases. After all, fewer releases means less manual testing, but this may increase Process Risk.

How do you decide whether to keep a test manual, or automate? The more automated a test is, the more cheaply it can be re-used. However, the process of automation can take longer, and so adds Schedule Risk. Whether or not it’s worth automating is to some extend going to depend on how much you value future time.

White-Box and Black-Box Testing

In the initial conception, Black-Box Testing ignores the implementation details of a component and tests the interface only.

White-box testing however considers the components within the box, and how they interact with one another in order to define the tests. This is fair enough if, for some reason, you are unable to test the components individually for some reason: knowing how something is implemented gives you an insight into where the bugs will hide, and therefore, where the risks lie.

Testing Level

However, if possible, it’s better to break open the white box and test the components themselves. This means you end up having “higher” and “lower” level tests, depending on the scope of the System Under Test. There are several advantages to this:

Expanding on this then, the Testing Pyramid idea is that lower level, automated tests which run quickly should be common, while there should be fewer of the more expensive “whole system” level tests.

Testing Pyramid

Finally, since manual tests are run by people (who are comparatively slow and costly), these should be the rarest kind of test.

Testing Team

Sometimes, testing is handled by external teams (possibly in other locales). This is often done as a cost-saving measure, but comes with some penalties such as:

Test-Driven Development

Also called test-first development, the idea here (from Extreme Programming) is that you write the tests before the code, in order that you think up-front about the requirements of the software you are writing. The aim of this is to minimize Complexity Risk via preventing developers from Gold Plating, and getting them to do The Simplest Thing That Can Possibly Work.

Additionally, by having test fail before they pass, you mitigate the risk of writing a “null” test (see below).

Code Coverage

Code Coverage tools are a useful way of showing you which parts of your software might contain bugs due to lack of testing, which is really useful in the Risk Evaluation phase of test-planning.

Sometimes code coverage spawns its own Map And Territory Risks though, where people forget that the goal should be mitigating overall project risk (via delivering functionality and so forth) and start to believe that the goal is delivering 100% code coverage. Writing tests to cover every get() method is a fools’ errand which increases the overall codebase complexity for no real reduction in Feature Risk.

Worse still is that having 100% code coverage does not guarantee an absence of bugs, or that the code will do what the users wanted it to do. Feature Risk is always there.

Risks Mitigated

There are so many different types of testing and this guide is not meant to be exhaustive. Instead, here is a table covering some of the main types of testing and the risks they mitigate:

Risk Mitigation
Boundary Risk System Integration Testing
CI Deployment
User Acceptance Testing
Dependency Risk Integration Testing
System Integration Testing
Production Risk Performance Testing / Load Testing
Non-Functional Testing
Disaster Recovery Testing
Security esting
Smoke / Sanity Testing
Software Risk Unit Testing
Component Testing
End-To-End Testing
Functional Testing
Feature Risk Browser-Based Testing
Accessibility Testing
Acceptance Testing (UAT)
Beta Testing
Visibility Risk Usability Testing
Corridor Testing
Complexity Risk Unit Testing
Automated Acceptance testing
Integration Testing

Attendant Risks

Firstly, it can be easy to fool yourself with tests: just because your tests pass does not mean your code is perfect. Vigilance is required against Map And Territory Risk:

Second, Testing is a double-edged sword. While it allows you to mitigate various Feature Risks, by adding test-code to your project you are necessarily increasing the complexity. Maintaining tests is hard work, and if you’re not careful, running tests can take time and slow down builds and add delay through Process Risk.

Third, if you are exploring functionality in order to flush out requirements, understand user behaviour or figure out performance characteristics, then there is no point in building tests yet: what you are doing is exploratory at best and the extra code will slow you down.

For these reasons, focus on writing the smallest number of tests that mitigates the risks.

See Also

Risk Based Agile Testing by Martin Ivison, which covers a lot of this ground in much more detail.

Continuous Integration

referred to.

Risk-Adjusted Value

Actually, this

How does this help?

real options

Found this interesting? Please add your star on GitHub to be invited to join the Risk-First GitHub group.