You leave an agent working on your strategy overnight. In the morning it has tested 60 variants, and the best one shows a Sharpe of 1.9 with a clean equity curve. You look at the curve. You don’t ask how many variants came before it, or how many it would take to find something that good by pure luck.
The agent made your research faster. It also made your false positives cheaper.
Over the last two issues we looked at the model as a trader, then as a research assistant that has already read the future. This one is about the agent as a search engine: what it really saves you, why every extra backtest raises the bar, and the one failure that no amount of statistics will catch.
1. The real gain is time, and nobody serious claims more
The firms that use agents most heavily describe the same benefit: throughput. Code gets written faster, papers get summarised faster, more ideas get tested in a week. None of them claims that the agent produces alpha on its own.
Man Group is the clearest example. Its agent proposes trading signals, and those signals face exactly the same thresholds as ideas from human researchers. The firm reports how many were approved. It does not report how many were tried, and that second number is the one that tells you how much luck went into the first.
2. Every extra backtest raises the bar
Test enough strategies with no edge at all and one of them will look good. That is not a flaw in your code, it is arithmetic. To see how fast it happens, I simulated strategies with zero true edge on five years of daily returns and tracked the best Sharpe found as the number of tests grows.
After about 45 tests, a Sharpe of 1 is what you should expect from noise alone. That matches the bound Bailey, Borwein, López de Prado and Zhu gave a decade ago: with five years of data, no more than about 45 independent configurations before an in-sample Sharpe of 1 means nothing. An agent working overnight crosses that line before breakfast.
A recent working paper by Gençay shows what this looks like with a real agent. A GPT-4.1 agent designed 100 strategies, with every attempt logged. Its best one scored a Sharpe of 1.69 during design, failed the Deflated Sharpe test once the 100 attempts were counted, and delivered 0.18 out of sample.
A Sharpe of 1.9 after 3 tries and a Sharpe of 1.9 after 60 are not the same finding.
3. Counting trials fixes selection, not leakage
The answer to section 2 is a registry: every backtest the agent runs gets logged, and the Deflated Sharpe Ratio is computed with that count. It works, and it is the first thing I would put in place.
It has a blind spot. In the same paper, Gençay plants a deliberate look-ahead leak in one strategy. It reaches a Sharpe of 34.7 and passes the Deflated Sharpe test with a perfect score. The statistics correct for how many times you searched. They cannot see that the code is reading tomorrow’s data.
That risk grows with agents, because they write most of the code. A normaliser computed over the full day, a feature shifted the wrong way, and the test passes while the backtest lies. That part is caught by reading and testing the code, not by any p-value. The registry also misses the ideas you dismissed in your head before the agent ever ran them.
Next time an agent hands you its best result, ask for the denominator before you look at the curve.
The registry, the sealed test window and the code checks are part of the process I run with my own agents, and this newsletter only shows pieces of it.
The complete workflow is in AI QuantPod.