More tests can mean more luck

The result you did not see

A backtest shows how a trading rule would have behaved on past data. But the best result in a long search can tell a misleading story.

Researchers David Bailey and his co-authors showed this with random data. Their software searched for a rule that looked good on one sample. On a separate sample, that rule often did poorly.

This is overfitting: a rule fits quirks in old data rather than a pattern that holds up elsewhere. Looking only at the winner hides all the failed attempts.

Where AI fits in

Our takeaway: asking AI for more versions does not make the best version stronger evidence. The paper studied computer searches, not today's AI assistants. The link to AI is our interpretation of the same selection problem.

Use AI to help explain a rule and document changes. Keep the test history, including the ideas you dropped. A polished explanation is not evidence that a strategy will work.

Keep a record of the search

Before you change a rule, write down what you expect the change to do. After the test, save the result beside that note. Include the data period, settings and trading costs.

Keep some data aside for a later check. If you keep changing the rule after seeing that check, the data is no longer untouched. The researchers also warn that a holdout alone does not account for all the trials in a search.

Next time you compare results, ask: how many ideas did we try before we chose this one? That is a better starting point than looking at the winner alone.