Choosing a useful test is not about using the newest tool. It is about obtaining evidence strong enough and relevant enough for a decision, with reasonable cost, time, and exposure. Strategyzer compares experiments by cost, setup time, run time, and evidence strength; these categories guide selection but vary across contexts [R04].

Start with the decision, not the method

Write: “If we find this evidence, we will ___; otherwise, we will ___.” If nothing would change, the test may not be necessary yet. Next identify whether you want to understand a need, observe behavior, see whether someone pays, verify technical capacity, or evaluate an economic condition. Exploratory objectives call for different methods from studies aiming to estimate rates precisely [R07][R17].

Before selecting a technique, write a testable hypothesis specifying the population, context, behavior, and time frame. The method should address the missing evidence, not substitute for a clearly stated claim.

Methods and the limits of their evidence

Test Main use What it does not establish alone
Discovery interview Understand experiences and motives Future purchase
Contextual observation Watch current tasks and barriers Population frequency without proper design
Survey Collect comparable self-reports Actual behavior from statements
Landing page Interaction with a message and offer Retention or payment without a transaction
Controlled advertisement Response to a particular offer and channel Overall profitability or product–market fit
Task prototype Understanding and use of a concept Operation at scale
Transparent presale Financial commitment under stated terms Future operational fulfillment
Real pilot Usage, delivery, failures, and costs in a bounded setting Universal scalability
A/B test Compare variants under controlled conditions Causality with incomparable groups or conditions

Presales and advertising create duties of transparency and delivery: do not deceptively imply unavailable services are ready [R10]. For testing materials, see prototype types.

If you need to reconstruct experiences and needs before testing an offer, the discovery-interview guide covers recruiting, non-leading questions, and the limits of self-reports. Interviews alone do not establish that anyone will buy.

Compare cost, setup, runtime, and evidence strength

A cheap test may involve difficult recruiting; a seemingly fast test may produce weak evidence. Estimate direct costs (tools, advertising, incentives), team hours, time to result, access to the segment, and ethical or operating risks. Do not treat “high” and “low” labels from another industry as universal prices.

Illustrative booking-service matrix:

Method Assumed budget Preparation Execution Main signal
Eight interviews S/240 3 days 4 days Context and problems
Landing page S/350 5 days 7 days Requests by channel
Three-restaurant pilot S/1,500 10 days 14 days Usage, errors, costs, continuity

These figures are invented to illustrate decisions, not real project estimates.

A practical selection tree

  1. Need to discover what happens today? Use observation and qualitative research.
  2. Need to know whether people understand or can use a proposed solution? Use prototypes and representative tasks.
  3. Need evidence of commitment? Present a clear offer, request, presale, or pilot, depending on risk and ethics.
  4. Need quantitative comparisons? Design comparable conditions and samples before using A/B testing.
  5. Need to verify delivery? Run a controlled pilot with recorded costs and incidents.

This is an editorial decision guide, not a Strategyzer-published decision tree. At each stage, ask whether the evidence would suffice for the decision. One test may lead to another: interviews → offer test → paid pilot, each with its own hypothesis and criteria.

Illustrative mapping of research questions to experiment methods.
Explanatory diagram: Evidence needed should drive cost and complexity. This is an original Atlas teaching illustration; the relationships shown do not replace an actual study.

What is the smallest sufficient test?

It does not always mean the cheapest. A click-through rate might justify investing in a pilot, not hiring a permanent team. Compare the cost of testing with the potential cost of making an uninformed decision. Difficult segment access adds recruiting costs. The full protocol appears in designing an experiment; risk priorities appear in critical hypotheses.

What you should have by the end: a matrix of three candidate tests, estimated cost and timing, expected evidence, limitations, and a justified method choice. Main sources: [R04], [R07], [R17], [R18].

Bibliography and references