A useful experiment begins well before meeting the first participant. Establish what you need to learn, from whom, under what conditions, and which finding would change the decision. Strategyzer's Test Card keeps hypotheses, tests, measures, and thresholds explicit. GOV.UK recommends planning objectives, recruitment, materials, and practice sessions [R03][R20].

1. Define the decision and a bounded hypothesis

Write a question that matters now: “Will eligible managers use a booking log for fourteen days?” State what investment depends on the answer. Link the claim to the hypothesis writing guide. Before testing, decide what evidence would support, weaken, or leave the claim unresolved.

If you have not yet chosen a way to gather that evidence, see how to select the right experiment based on the decision, cost, time, and strength of the expected signal. Choose the method before detailing its protocol.

2. Recruit the people who matter

Define population, inclusion and exclusion criteria, location, and contact method. A restaurant booking service needs people who manage reservations, not only owners who never use the system. Document whether recruiting is convenient, experience-based, or probability-based. Convenience samples can produce useful insights but seldom justify sweeping population claims [R07][R08].

There is no magic participant count. Qualitative interviews, usability studies, surveys, and A/B tests have different objectives; statistical inference calls for a defined design and precision. GOV.UK provides guidance on methods and participant numbers but no universal sample size for all decisions [R07].

3. Design tasks, materials, and conditions

Specify the offer, steps participants see, device, instructions, facilitator, and what may vary. For comparative trials, make conditions comparable; consider group or task-order randomization when appropriate. NIST covers factors, randomization, and confounding in formal experimental design, but this does not mean every exploratory interview needs to become a laboratory trial [R11].

Before finalizing the conditions, review how to recognize and reduce experimental bias: participant selection, leading instructions, and variables changed together can distort comparisons. Safeguards should fit the question and the method.

If using a prototype, verify it supports the relevant tasks and make clear what is simulated versus functional.

4. Predefine metrics, denominators, and criteria

“Measure interest” is not specific enough. Define an event: request submitted, session completed, booking recorded, critical error, actual payment, or cancellation. Keep denominators: of 20 people contacted, how many qualified, received the offer, accepted, and completed? Document the period, measurement method, and decision to be made under different outcomes.

Element in a hypothetical pilot Planned definition
Hypothesis Five of ten eligible restaurants will accept a pilot
Population Managers responsible for telephone reservations
Procedure Standard demonstration and identical fourteen-day offer
Primary metric Acceptances / offers delivered to eligible contacts
Secondary metrics Weekly use, errors, booking time, support
Threshold 5/10, an example requiring business justification
Inconclusive outcome Insufficient recruitment or changed conditions

The threshold is hypothetical; acceptance by 5/10 does not mean half of all restaurants will buy. Accepting a free pilot also does not establish future payment.

Six key elements of an experiment protocol before execution.
Explanatory diagram: Make decisions explicit before collecting outcomes. This is an original Atlas teaching illustration; the relationships shown do not replace an actual study.

5. Plan budget, duration, and responsibilities

Allow time to recruit, confirm consent, prepare the artifact, train staff, execute, analyze, and document. Specify a decision owner and, where possible, a separate recorder. Define stopping criteria for serious technical failures, participant risks, or commercial promises that cannot be fulfilled.

The protocol's budget, duration, and owners should be carried into a realistic schedule with dependencies, availability, and float. A test requiring unavailable equipment, approvals, or participants is not ready simply because a start date has been entered.

6. Practice before the real test

Run through the protocol; check timing, instructions, logging, and artifact functionality. A pilot may reveal leading questions or tasks that fail to measure the hypothesis [R20][R18]. If you change a quantitative protocol after starting, version the change and avoid pooling incompatible observations as though they were identical.

Explain who is researching, purpose, voluntary participation, recording, data usage, and retention. Do not present a presale as an available product if you cannot fulfill it; explain relevant conditions and refunds. Specific legal duties vary by jurisdiction; GOV.UK guidance offers research practices rather than universal legal advice [R10][R16].

If the test involves personal data, payments, preorders, or a regulated activity, use the guide to identifying applicable legal and operational requirements before running it. Do not infer specific legal duties from guidance written for another jurisdiction.

Protocol template

ID · version · blocked decision · hypothesis · risk category · recruitment criteria · method · tasks · materials · metrics/denominators · thresholds · duration · consent · owners · incidents · stopping conditions · expected limitations. Compare the finished record with objective results documentation to complete the cycle.

What you should have by the end: a protocol another person can understand and run without redefining the objectives. Main sources: [R03], [R07], [R08], [R10], [R11], [R18], [R20].

Bibliography and references