Curriculum/multiple-testing
Multiple Testing
statistics·L1 · combinator·stub
Replacesthe belief that finding a strategy with p<0.05 means it works.
Run 100 backtests; 5 will hit p<0.05 by chance alone. The fix is correction for the number of trials — Bonferroni (conservative, family-wise error rate), Benjamini-Hochberg (false-discovery-rate control), or held-out validation. Without correction, every quant team produces 'great' backtests at the rate the statistician predicted.
Prerequisites
Unlocks
Bridges
- backtest-overfittingshared failure modeBacktest overfitting is multiple testing in disguise — the trial count is implicit (every parameter combination tested) and usually unmeasured. Bailey & López de Prado's *Probability of Backtest Overfitting* (PBO) turns the implicit count into a measurable trial-count adjustment.
- factor-zoo-replication-crisisshared failure modeHarvey, Liu & Zhu's *…and the Cross-Section of Expected Returns* applies multiple-testing correction to the 300+ factors in the academic literature. After correction, the majority do not survive — the same family-wise failure mode at literature scale.
This concept is a node in the curriculum DAG. The full lab — page blocks, done state, references — has not been authored yet. The relations above describe where it sits in the graph.
Author at: content/concepts/multiple-testing/card.ts