An experiment can be perfectly documented and still produce misleading conclusions. Bias may enter through participant selection, wording, test conditions, recording, and interpretation. Reducing bias does not eliminate uncertainty: it means identifying what could distort a decision and designing proportionate safeguards [R08][R11][R17].
Bias, random error, and measurement failure
Bias systematically shifts results toward a conclusion; random variation changes across samples; measurement error occurs when an instrument records the wrong event. They do not all have the same remedy: a huge sample cannot repair the systematic exclusion of particular users.
Risks that appear before a test starts
- Confirmation bias: seeking only evidence consistent with the preferred idea.
- Selection bias: recruiting only friends, nearby businesses, or enthusiasts.
- Self-selection: volunteers may differ from people who ignore an invitation.
- Survivorship and attrition: analyzing only users who finish the pilot.
- Social desirability: participants say what they think researchers want to hear.
- Leading questions: “How much time would this excellent tool save you?” suggests an answer.
- Observer effects: facilitator presence or behavior changes what participants do.
Combine relevant criteria and recruitment channels, and document who was excluded [R08]. The broader article on evaluating evidence without bias covers additional issues; here the focus is safeguards during experiments.
An example that changes three variables at once
Version A is offered to large restaurants in the morning at S/80 with a time-saving message. Version B is offered to small venues in the afternoon at S/120, emphasizing ease of use. If B attracts fewer requests, we cannot distinguish price, venue size, time of day, and message effects. This is confounding: several differences compete as explanations.
In comparisons intended to estimate effects, keep other conditions reasonably similar, document variables, and consider random assignment when suitable. NIST explains experimental designs, blocking, and confounding. In qualitative tests, changing questions can be useful for discovery, but it must be recorded rather than merged into a single numerical comparison [R11][R17].
Define the criteria before seeing outcomes
Set metrics, denominators, thresholds, and duration before testing; version the protocol. Repeatedly watching a metric and stopping when it looks best can exaggerate the signal. If criteria change after seeing data, label the new work exploratory rather than pretending it confirmed the original criterion. See experiment design.
Account for dropouts, records, and negative results
At the end ask how many were contacted, eligible, started, and finished. If four of ten venues drop out, do not report satisfaction solely among the six survivors without explaining the restriction. Retain incidents, unusual cases, and competing explanations; “did not respond” and “declined” are not necessarily the same thing.
To preserve these distinctions throughout the test, use the guide to objective results documentation. Tracking eligible participants, dropouts, missing events, and protocol changes helps prevent reconstructing denominators after seeing the outcomes.
Ethics and privacy also affect data quality
Pressured participants do not offer reliable responses. Explain purpose and recording, avoid unnecessary personal information, and limit access and retention. Check the legal framework applicable to the study's location and nature. GOV.UK offers guidance for its context, not universal legal rules [R10][R16].
A stage-by-stage checklist
| Before | During | After |
|---|---|---|
| Predefine hypothesis and metrics | Keep instructions consistent | Count attrition and exclusions |
| Recruit suitable profiles | Record changes and incidents | Compare against original threshold |
| Avoid leading questions | Do not suggest answers | Report contradictory cases |
| Plan proportionate controls | Maintain consent | Record alternative explanations |
What you should have by the end: a bias risk register with controls, limitations, and owners, ready to support results analysis. Main sources: [R08], [R10], [R11], [R16], [R17].
Bibliography and references
- [R08] GOV.UK Service Manual. Finding participants for user research. https://www.gov.uk/service-manual/user-research/find-user-research-participants — Reclutamiento, pertinencia y sesgos de selección.
- [R11] NIST/SEMATECH e-Handbook. Choosing an experimental design. https://itl.nist.gov/div898/handbook/pri/section3/pri3.htm — Variables, aleatorización, factores de confusión y diseños comparativos.
- [R17] Nielsen Norman Group. Quantitative vs. Qualitative Usability Testing. https://www.nngroup.com/articles/quant-vs-qual/ — Investigación cualitativa y cuantitativa.
- [R10] GOV.UK Service Manual. Getting informed consent for user research. https://www.gov.uk/service-manual/user-research/getting-users-consent-for-research — Consentimiento y transparencia en investigación.
- [R16] GOV.UK Service Manual. Managing user research data and participant privacy. https://www.gov.uk/service-manual/user-research/managing-user-research-data-participant-privacy — Protección y retención de datos.