Backtesting pitfalls: overfitting, costs and survivorship
How a backtest can look excellent and still fail live, and the checks that expose the problem before you risk money.
Overfitting
Overfitting means tuning a strategy so closely to past data that it captures noise rather than a repeatable edge. A rule set with many parameters can fit almost any history. Warning signs include results that depend on one exact setting, and performance that collapses when the period or pair changes slightly.
Costs, slippage and look-ahead
Backtests often use mid prices, zero spread and perfect fills. Real trading pays the spread, faces slippage in fast markets, and pays swaps on held positions. Look-ahead bias happens when a rule uses information that would not have been available at the time, such as a daily close used for an intraday decision. Check every rule for timing, and add realistic costs.
Survivorship and sample size
Survivorship bias occurs when you test only instruments, brokers or traders that still exist today. A small number of trades can produce a dramatic equity curve by chance. Use a long enough history with different market conditions, and judge results by their risk as well as their return, including the worst drawdown.
Key takeaways
- Be suspicious of rules tuned to one exact setting.
- Include spreads, slippage, swaps and realistic fills.
- Test on out-of-sample data, over long periods, and judge the worst drawdown.