You ask a language model to help you build a crypto momentum strategy. Start of 2022, ten liquid coins, nothing exotic. It gives you a clean list with a sensible reason for each pick, you run the backtest, and the result looks solid.
What the model doesn’t mention is that it already knows how 2022 ends. It knows which top-ten coin collapsed in May, and which exchange token went to zero in November. Nobody asked it to use that knowledge. Nobody asked it not to, either, and it couldn’t comply if you did.
The model doesn’t need to trade to contaminate your backtest. It only needs to help you choose.
Last week we looked at what happens when you hand a model the trading itself. This issue looks at the quieter case, the one that affects anyone using an LLM in research: what the model remembers, how that memory leaks into a backtest it never touches, and where the bias turns out to be smaller than you would expect.
1. It doesn’t forecast the past, it remembers it
A language model is trained on text up to a fixed date, its training cutoff. Ask it about anything before that date and it is not predicting. It is recalling.
Lopez-Lira, Tang and Zhu measured this directly with GPT-4o, whose training data stops in October 2023. They asked it a simple question, quarter after quarter: would US GDP growth come in above 2.5%? Then they told the model to ignore everything it knew after 2010, as if it were living in 2010 and forecasting forward.
The instruction changed nothing. The model was right 97.6% of the time before 2010, and 98.0% of the time on the period it had been told to forget. Only after its real cutoff, on quarters it had never read about, did accuracy fall to 40%.
A prompt cannot create a cutoff. The model can’t unlearn what it read, and it won’t tell you when it is using it.
The 40% rests on a handful of quarters, so it is not a precise score. What it shows is the break: near-perfect while the model knows the answer, ordinary the moment it doesn’t.
2. It contaminates the backtest without ever placing a trade
Most people who use an LLM in research never let it trade. They let it help: suggest a universe, propose features, pick a sensible test window. Each of those is a choice made before the backtest runs, and each is a place where knowledge of the outcome can slip in unnoticed.
I tested the most obvious case. On 1 January 2022, Terra (LUNA) was the ninth largest crypto asset, worth about $33 billion. Five months later it was gone. I asked Claude ten times, in fresh sessions, for the ten largest coins as of that date. LUNA was in every answer, and every list matched the real ranking coin for coin.
That is reassuring, and it is also memory. The model didn’t estimate the ranking, it recalled it. Asked for a fact, it returned the fact. The exposure starts when you ask for judgment instead: which coins deserve a place, which features look promising, which period is representative. There is no ranking to recite, and what the model knows about the outcome can shape the answer without leaving a trace.
That effect has been measured. Li et al. backtested GPT-4o trading agents on NASDAQ-100 stocks in 2021, then ran them again in late 2024, after the model’s cutoff. The market rose about 13% in both windows. The agents’ Sharpe ratios fell by 51% to 62%.
The leak is not in your data or your code. It is in the choices you made with the model’s help.
3. The bias is uneven, and the rule is simple
Not every use is equally exposed. The bias is large when the model recalls facts or makes decisions, because both draw directly on what it remembers. It is much smaller when the model only turns text into a score. He, Lv, Manela and Wu trained models only on text available at each date and compared them with a modern LLM on news-based return prediction: the portfolios performed comparably, and the authors call the look-ahead bias in that setting “relatively modest”.
So the danger sits in the use, not in the tool. The rule I apply:
- never use the model as a source of prices, macro data or historical facts;
- never ask it what would have worked over a period it has already read;
- when it helps choose a universe, features or a test window, judge the result only on data after its training cutoff.
That last rule has a cost. The window after the cutoff is short, often a few months, so a Sharpe measured there carries a wide error band, and it closes again with every new model release. It keeps the test honest. It doesn’t make it powerful.
Before you trust a result the model helped you build, look up one date: its training cutoff. If your test ends before it, the model may have chosen with the answers in hand.
I use these guardrails every day, and this newsletter only shows pieces of how. The complete process, step by step, is in AI QuantPod. The first three sections are already online, and the pre-sale price of $397 ends on November 9.
If I sign up for the ai pod, how long do I get access to discord (and specifically your 1on1 help)?