Atlas of Cognitive Biases #57 · Data and testing. A perfect backtest, a poor future
What it looks like in trading
parameters chosen to make the historical result as good as possible
testing hundreds of variants and picking the best
rules like “but not on Thursdays after 1:00 PM”
An example from a session
A simulation from Biases by the numbers #6: 200 random “strategies” on random data. The best one has a great result in the test period and is average in the next. No surprise: it never had an edge.
What the research says
Halbert White (2000) proposed a test of whether the best of many tested strategies is better than chance. Bailey, Borwein, López de Prado and Zhu (2014) showed how easy it is to get a falsely great backtest when you try many variants. For this reason Harvey, Liu and Zhu (2016) proposed stricter significance thresholds in finance.
Fitting a method to the details of historical data, including random noise. Such a method describes the past brilliantly and works poorly in the future. Also known as data snooping.
How does overfitting show up in trading?
Typical signs: parameters chosen to make the historical result as good as possible; testing hundreds of variants and picking the best; rules like “but not on Thursdays after 1:00 PM”.
Educational content, not psychological advice or an investment recommendation.