Design5 min read

User Testing

Validate design decisions with real users before shipping - through moderated sessions, unmoderated tests, prototype walkthroughs, and A/B experiments.

User TestingUsabilityA/B TestingPrototype TestingUX Research

skill: User Testing category: Design tags: [User Testing, Usability, A/B Testing, Prototype Testing, UX Research] readingTime: 5

User testing is the most cost-effective quality assurance for UX - it's cheaper to discover problems with a prototype than to fix a shipped product. 5 users in a moderated session uncover 85% of usability issues. Unmoderated tests at scale complement but don't replace qualitative insight.

Workflow

Choose the right testing methodDifferent methods answer different questions - moderated sessions explore why; unmoderated tests measure task success at scale; A/B tests compare conversion metrics.
Write effective test tasksVague tasks produce vague results. Realistic, specific tasks close to actual usage scenarios reveal whether the design works in context.
Run unmoderated tests with Maze or LyssnaUnmoderated tests collect quantitative task data from 30-100 users asynchronously - scalable but can't probe why users failed.
Run A/B tests for quantitative decisionsA/B tests answer 'which version converts better?' with statistical confidence - removing gut-feel and HiPPO decisions from high-stakes UI choices.
Synthesize and prioritize findingsRaw test observations aren't actionable. Severity scoring helps teams prioritize which issues to fix in the next sprint.

Output Format

User test report: tasks + metrics, key issues with severity, root causes, and design recommendations.

Test Report:
├── Overview: method, participants, date
├── Task metrics: completion rate, time on task
├── Issues:
│   ├── CRIT-01: 4/5 users missed the submit button
│   ├── MAJ-01: Payment form error messages unclear
│   └── MIN-01: Help tooltip too small to read
└── Recommendations with priority

Examples

Input

Output

Edge Cases & Gotchas

Sample ratio mismatch in A/B tests - If A gets 5,000 users and B gets 3,000, the traffic split failed. Investigate the assignment mechanism before concluding on results.
Hawthorne effect - Users behave differently when they know they're being tested. In moderated sessions, minimize interviewer cues; in unmoderated, keep the recruitment message neutral.
Testing with wrong users - Testing a B2B enterprise feature with consumers gives irrelevant data. Screen participants rigorously - job title, tool usage, company size.
Multiple testing problem - Testing 10 metrics simultaneously has a high false positive rate. Define one primary metric before running. Secondary metrics are exploratory only.

When to Break These Rules

Skip A/B testing for features with no measurable conversion impact (visual refinements, copy edits on low-traffic pages) - the variance in the test is larger than the effect.
Ship without testing for time-critical fixes (security patches, critical bugs) - but do a rapid 3-user session the next day to verify the fix doesn't introduce new UX issues.

Reference Files

https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
https://www.evanmiller.org/ab-testing/sample-size.html