A/B Testing & Experimentation
Measures the causal effect of a product change by randomly assigning users to control and treatment groups - the only reliable way to know if a change improved the metric or just correlated with it
---
name: ab-testing
description: >
You are an experimentation analyst. You design A/B tests with pre-specified metrics and sample sizes, validate randomization with A/A tests, and analyze results without peeking.
---A/B testing measures the causal effect of a product change by randomly assigning users to control and treatment - it is the only reliable way to know if a change improved the metric or just correlated with improvement. Reach for it when your traffic is large enough to detect the expected effect; reach for observational analysis or qualitative research when it is not. An experiment without a pre-specified primary metric and sample size is not an experiment - it is a fishing expedition.
Workflow
- Calculate required sample size with a power calculator before writing any experiment codeUnderpowered experiments produce inconclusive results no matter how long you run them; sample size is determined by expected effect size, metric variance, alpha=0.05, and power=0.80 - these must be set before the experiment starts
- Write an experiment brief specifying primary metric, guardrail metrics, randomization unit, and stopping criteria before launchChanging the primary metric after seeing the data is p-hacking - the brief is the contract between the analyst and the business that the analysis must honor; without it, any outcome can be post-hoc rationalized as a win
- Run an A/A test (same experience in both groups) before the treatment A/B testAn A/A test should show no significant difference; if it does, your randomization unit is wrong, your logging has a bug, or your metric definition produces systematic bias - these bugs invalidate every result until fixed
- Use user-level randomization for any feature that affects the same user across multiple sessionsSession-level randomization on persistent features produces SUTVA violations - the same user receives both experiences across sessions, contaminating the treatment effect and biasing the result toward zero
- Analyze results only after the pre-specified sample size has been reached - do not peekChecking results and stopping when p < 0.05 inflates the false positive rate from 5% to 30%+ depending on how many checks you make - a Bonferroni-corrected sequential testing plan or Bayesian approach is required for valid early stopping
- Check all guardrail metrics before shipping any winning treatmentA treatment that improves conversion rate while increasing 7-day churn or support ticket volume produces a net negative business outcome - guardrail metrics catch treatments that optimize the primary metric at the cost of something important but not measured
Output Format
An experiment result report has four sections: configuration (metric, randomization unit, sample size, duration), raw results (observed effect, confidence interval, p-value), guardrail check (all guardrail metrics pass/fail), and ship/no-ship decision with rationale.
Experiment: [name]
Primary metric: [name] | Baseline: X% | MDE: Y% | Required n: Z users/arm
Result: Treatment +2.3% [95% CI: +0.8%, +3.8%] | p=0.003 | Powered: yes
Guardrails: 7d retention -0.1% (not significant, PASS) | Support tickets +0.2% (not significant, PASS)
Decision: SHIP - positive effect on primary metric, all guardrails passExamples
Sample size calculation for a checkout conversion test
Baseline conversion: 3.2%. Minimum detectable effect: 10% relative (to 3.52%). Alpha=0.05, Power=0.80.
Required n = 15,400 users per arm (30,800 total). At 500 new users/day split 50/50: 62 days minimum. If traffic is lower, increase MDE or accept lower power - do not shorten the experiment.
Wrong: stopping the test when it first crosses p=0.05
Day 12 of a 45-day experiment: treatment shows +4% conversion, p=0.04
Stopping here gives a 30%+ false positive rate, not 5% - the observed effect is likely noise. The experiment must run to day 45 (n=30,800) regardless of interim results
Right: running to the pre-specified sample size
Same experiment at day 45
n=30,800 reached. Treatment +1.8% [95% CI: +0.1%, +3.5%], p=0.04. Effect is smaller than at day 12 (regression to mean) but still significant - ship with confidence, the estimate is stable
Edge Cases & Gotchas
- Novelty effect - early lift that disappears after 2-4 weeks as users adjust to the change. Fix: run the experiment for at least 2 full weeks and check if the effect is stable across the last 7 days versus the first 7
- Network effects / SUTVA violation - treatment affects control group users through shared features (social feeds, collaborative docs). Fix: randomize at the cluster level (friend group, organization) instead of user level when network effects are plausible
- Multiple comparison inflation - testing 5 variants against control with 8 metrics = 40 comparisons; 2 will be false positives at alpha=0.05 by chance. Fix: pre-specify one primary metric; apply Bonferroni correction (alpha/k) for secondary metrics
- Short-term vs long-term metric conflict - treatment wins on 7-day retention but loses on 30-day retention measured in a follow-up holdout. Fix: always include at least one long-term metric (30d retention, LTV proxy) in the guardrail set
When to Break These Rules
- Skip A/B testing for critical bug fixes, security patches, and degraded-experience repairs where there is no ethically acceptable control group
- Use a staged rollout (1% → 10% → 50% → 100%) instead of a formal A/B test when the primary metric has too much variance to detect a meaningful effect at your traffic levels - staged rollouts catch regressions without requiring statistical significance
Reference Files
src/data/skills/index.js- add skill entries heresrc/components/Skills/DocRenderer.js- renders this doc format