A useful test does not start with clicking buttons. It starts by defining the condition to demonstrate, what could fail, and what evidence would count.
During development we need to establish that what we build works under relevant conditions. That includes functions, integrations, security, accessibility, material strength, and the ability to deliver a service consistently. The Pillar 3 business experiments evaluate hypotheses about customers, demand, and the model. Solution testing verifies behaviors and qualities of a specific version. These activities are connected but not interchangeable. UK government technical guidance recommends testing usability, functionality, capacity, security, and accessibility according to service risks [R01].
1. Distinguish verification, validation, and business outcomes
Verification: Does the solution satisfy a stated requirement? For example, does a changed booking retain the new time? Validation: Does it meet the need in the real context? For example, can the host understand the change and avoid errors? Business outcome evaluation: Does that improvement affect retention, costs, or revenue once the solution is used? Different claims require different evidence.
A functional test can pass while the experience remains confusing; a small usability test can succeed without demonstrating security; inspecting a physical component does not establish durability in all conditions. State exactly what a test supports.
If the question concerns a business hypothesis rather than only how the built solution works, see how to design rigorous experiments. Verifying a feature does not by itself demonstrate demand or willingness to pay.
2. Start with requirements, risks, and scenarios
Reuse P5-04's requirements and acceptance criteria, together with P5-07's quality decisions. Build traceability requirement → risk → test case → expected outcome → evidence.
For instance, the requirement “avoid duplicate bookings” suggests testing simultaneous requests for the same slot, a network interruption, accidental repeated submission, and behavior after recovery. These cases cannot prove the absolute absence of defects, but they examine failures that threaten core value.
Do not invent cases merely to increase test counts. One hundred happy-path tests that overlook the critical failure may offer false confidence.
Test selection requires a prior definition of acceptability: the quality plan connects attributes, risks, and criteria so a team does not approve a solution merely because many test cases ran.
3. Select test types based on the product and risk
| Type | Question | Example |
|---|---|---|
| Functional | Does a rule or action work as specified? | Create and change a booking |
| Integration | Do components exchange information correctly? | Calendar and notifications |
| System / end to end | Does the complete flow work? | Request, confirmation, change |
| Regression | Did a change break previously working behavior? | Create bookings after editing changes |
| Performance and load | Does it respond under relevant demand? | Concurrent operations |
| Security | Are resources and data appropriately protected? | Unauthorized access to another booking |
| Accessibility | Can people use it with different abilities and aids? | Keyboard and assistive technology |
| Usability | Can people complete and understand a task? | Change a booking without help |
| Operational | Can staff deliver and recover service? | Handle changes during a busy shift |
Every project does not need every test. Some tests require specialists, equipment, or accredited methods. Automation helps repeat measurable checks but does not replace user observation or expert assessment of complex concerns [R01][R02].
4. Cover physical products and services, not software alone
For physical products, consider dimensions, fit, strength, wear, temperature, environmental conditions, handling, packaging, and safety where relevant. Identify methods, instruments, samples, tolerances, and qualified personnel. Do not declare a safety-critical product conformant after a visual check alone.
For services, test the real sequence: request, assignment, preparation, delivery, exceptions, and recovery. Include varying shifts or workload scenarios when appropriate. A tabletop simulation may expose unclear instructions; an actual pilot may reveal capacity limits that were invisible in planning.
For hybrid solutions, test interfaces between components: the system records an order, but does it reach the employee, can the message be understood, and what happens if the digital system goes offline?
5. Write reproducible test cases
Each test should specify an ID, requirement and version, purpose, preconditions, data or materials, steps, expected outcome, environment, owner, and evidence to retain. Add the observed result, status, and linked defects after execution.
| Field | TC-02 — time conflict |
|---|---|
| Precondition | Booking A confirmed for a specific time |
| Action | Another person attempts to confirm the same resource and time |
| Expected result | Conflict explained without changing booking A |
| Evidence | Action log and final state of both requests |
| Blocking condition | Do not approve version if incompatible bookings both confirm |
This is illustrative, not a completed production specification. A real project must define what constitutes a conflict, what resources have shared capacity, and how changes are synchronized. A test that passes without checking the final state may be incorrectly designed.
6. Prepare suitable environments, participants, and data
A test environment should represent what matters without exposing people or data unnecessarily. Prefer fictional or pseudonymized records and limited access where appropriate. Record version, device, browser, configuration, material, batch, date, and any differences from real use.
Observe people who did not design the solution when evaluating comprehension. Builders know shortcuts new users do not. For web accessibility, WCAG 2.2 contains testable criteria; automated tools should be complemented with manual evaluation and relevant assistive technology checks [R02][R03].
Do not copy participant counts or traffic levels from unrelated contexts as universal benchmarks.
7. Execute without quietly changing the criteria
Set expected results before executing. During testing, record events, failures, assistance, unusual conditions, and protocol changes. If a procedure needs to change, version it and identify which earlier results are no longer directly comparable.
Distinguish passed, failed, blocked, not run, and inconclusive. Not run is not passed. A blocked test caused by invalid data does not automatically prove product failure, but it leaves an evidence gap. An improvised fix during a session calls for a fresh run under known conditions.
8. Log errors, effort, time, quality, and consequences
The record should answer: what occurred, in which version, how to reproduce it, expected versus observed behavior, impact, observed frequency, evidence, and investigation owner. Add time, interventions, wasted materials, and customer consequences where relevant.
Do not confuse an observed incident with a confirmed root cause. “Confirmation was duplicated after two clicks” is an observation; “The user double-clicked because they are careless” is an unsupported interpretation.
P5-09 will cover finding classification and priority. This article focuses on producing reproducible evidence through test execution.
After producing a reproducible record, organize and prioritize findings to decide what to fix, investigate, or postpone. Test execution provides evidence; it should not automatically determine backlog priority.
9. Retest, run regression, and decide what blocks delivery
After a defect is fixed, rerun the failed case and related checks that the change may affect. Closing a ticket is not enough. In software, automated recurring regression checks can detect breakage; in manufacturing, new units or batches may need inspection; in services, repeating a process with another shift helps examine consistency.
Define blocking failures, permissible limitations, and who may accept residual risk before delivery. A security weakness or unmet mandatory obligation must not be hidden behind a “known issue” note. Pillar 4 addresses overall launch readiness; this test plan establishes quality evidence for the solution version.
Worked example: testing a booking system
A fictional team prepares a version with booking creation, editing, and cancellation. It designs four groups of cases: normal operations, time conflicts, permissions, and recovery after interruptions. It also observes hosts who were not involved in development.
In a hypothetical run, edits save correctly, but the screen does not clearly distinguish “Save” from “Close.” The persistence technical test passes, while the comprehension task fails for some participants. The team keeps both findings rather than erasing one to create an apparently consistent story.
After redesigning the confirmation message, it reruns the usability task, the persistence test, and the cancellation regression. It preserves versions, results, and conditions. It still cannot claim the solution is profitable or can serve hundreds of restaurants—those claims require other evidence.
Common mistakes and limitations
Testing only the happy path; executing without expected results; unnecessarily using actual personal data; confusing automation with usability; labeling “not run” as passed; changing criteria after seeing outcomes; deleting failures; skipping regression; extrapolating a low-load pilot to production; equating quality testing with commercial validation.
A test strategy should be proportionate to risk and intended use, with transparent acknowledgment of what was not tested.
What you should have when finished
A test plan and matrix linked to requirements, covering relevant types, preconditions, steps, expected and observed results, evidence, defects, owners, outstanding checks, and decisions on blocks or retests. A second team should be able to reproduce the checks without guessing their purpose.
References
- [R01] UK Government Digital Service. Quality assurance: testing your service regularly. https://www.gov.uk/service-manual/technology/quality-assurance-testing-your-service-regularly
- [R02] UK Government Digital Service. Making your frontend accessible. https://www.gov.uk/service-manual/technology/accessibility-for-developers-an-introduction
- [R03] W3C. Web Content Accessibility Guidelines (WCAG) 2.2. https://www.w3.org/TR/WCAG22/