Statistics for Data Science
Apply statistics to data problems - hypothesis testing, A/B test design, confidence intervals, regression, and Bayesian reasoning.
skill: Statistics for Data Science category: AI & ML tags: [Statistics, Hypothesis Testing, A/B Testing, Probability, Bayesian Inference] readingTime: 5
Most data science statistics mistakes are conceptual, not computational. p-values are misinterpreted, sample sizes are too small, multiple comparisons inflate false positives, and causation is inferred from correlation. The computational part is easy (scipy does it in one line); the hard part is designing the right test for the right question.
Workflow
- Understand distributions and probabilityThe right statistical test depends on the data distribution. Applying a test that assumes normality to heavily skewed data produces wrong conclusions.
- Design and analyze A/B testsA/B tests quantify the causal effect of a change. Underpowered tests miss real effects; multiple comparisons inflate false positives; stopping early biases results.
- Interpret p-values correctlyThe p-value is the most misinterpreted concept in statistics. Wrong interpretation leads to bad decisions at scale.
- Run regression analysisRegression quantifies relationships between variables and predicts continuous outcomes. Understanding regression assumptions prevents invalid conclusions.
- Apply Bayesian reasoningBayesian methods provide probability statements about hypotheses (what you usually want), incorporate prior knowledge, and update beliefs as evidence accumulates - no fixed sample size required.
Output Format
Statistics: shapiro normality test before parametric, proportions_ztest A/B with prior power analysis, Cohen's d effect size, statsmodels OLS with VIF, and PyMC Bayesian A/B with P(test>control).
Power analysis FIRST: calculate_sample_size(baseline, mde, alpha, power)
proportions_ztest([test_conv, ctrl_conv], [test_n, ctrl_n])
Cohen's d = (mean_b - mean_a) / std_a - effect size context
sm.OLS(y, X).fit(); VIF > 10 → multicollinearity problem
PyMC Beta prior + Binomial likelihood → P(test>control)Examples
Edge Cases & Gotchas
- Network effects in A/B tests - User-level A/B testing fails when users interact (messaging, social features) - control users see test features via interactions. Use cluster randomization (by team, org, or geographic region).
- Simpson's paradox - A treatment may appear better overall but worse within every subgroup (due to confounding). Always stratify analysis by key segments before concluding.
- Novelty effect - New designs initially get higher engagement due to novelty, not genuine improvement. Run tests for at least 2 full weekly cycles; include users exposed >7 days for post-novelty analysis.
- CUPED variance reduction - CUPED (Controlled-experiment Using Pre-Experiment Data) reduces variance using pre-experiment covariate (e.g., previous-week metric), increasing test sensitivity without increasing sample size.
When to Break These Rules
- Use Bayesian methods (PyMC, Stan) instead of frequentist tests when prior knowledge is available, early stopping is needed, or you want probability statements rather than binary reject/fail-to-reject decisions.
- Skip formal hypothesis testing for exploratory data analysis - EDA is for generating hypotheses, not testing them. Test hypotheses on new data, not the data you explored.
Reference Files
https://docs.scipy.org/doc/scipy/reference/stats.htmlhttps://www.pymc.io/projects/docs/en/stable/