Curriculum/backtest-pitfalls
Backtest Pitfalls
research process·L1 · combinator·stub
Replacesthe belief that look-ahead bias is the only backtest bug.
Three independent classes silently inflate measured performance: (1) look-ahead bias — data from t+1 leaks into t's decision; (2) survivorship bias — today's index members used for historical periods; (3) in-sample leakage — CV folds overlap in time and contaminate. All three look like 'great results' until the strategy goes live and the alpha vanishes.
Unlocks
Bridges
- factor-zoo-replication-crisisshared failure modeBacktest pitfalls at single-strategy scale and the factor-zoo replication crisis at literature scale are the same statistical failure mode — implicit multi-test counts unmeasured, in-sample leakage unpurged.
- live-trading-pnl-gapshared measurementThe size of the backtest-to-live PnL gap is the integrated cost of un-fixed pitfalls. A gap of more than 30% of expected return on the first month live is the standard signal that one or more pitfalls were unaddressed.
This concept is a node in the curriculum DAG. The full lab — page blocks, done state, references — has not been authored yet. The relations above describe where it sits in the graph.
Author at: content/concepts/backtest-pitfalls/card.ts