You give a frontier model live prices, a funded account and a clear instruction: maximise return, manage risk. No human in the loop, real money on the line. Seventeen days later, four of the six models have lost more than 40%. One is up 22%.
That was Alpha Arena, season one. The tempting reading is that some models simply trade better than others. The data doesn’t support it.
A language model given data and an account does not come with an edge. It reads the same market everyone else reads, and it decides differently every time you ask.
This issue opens a short series on AI in systematic research, and it starts with the question most people ask first: can you hand the trading to the model? We will look at what live AI trading results actually prove, why the problem is structural rather than a matter of picking a smarter model, and where an LLM genuinely earns its place in a research workflow.
1. Seventeen days of live trading prove almost nothing, in either direction
Alpha Arena gave six frontier models $10,000 each in real money to trade crypto perpetuals, with the same prompt and the same data. Qwen finished up 22%. DeepSeek scraped 5%. GPT-5 ended at $4,126, and Gemini and Claude both gave back more than 40%.
It looks like a ranking. Statistically, it is noise. Under the usual assumptions, the standard error of an annualised Sharpe ratio is roughly one over the square root of the test length in years (Lo, 2002). Seventeen days is less than 5% of a year. That puts the standard error near 4.6, and the 95% range of a strategy with no edge at all somewhere between -9 and +9.
A +22% and a -58% can come from the same absence of edge, with different leverage and different luck.
The uncertainty shrinks, but slowly. It takes years, not weeks, before a Sharpe ratio starts to mean something.
The uncertainty shrinks, but slowly. It takes years, not weeks, before a Sharpe ratio starts to mean something.
2. Trading is a process, and the model only covers its weakest step
A systematic strategy is a chain: a hypothesis about why prices should move, a signal that measures it, a sizing rule, risk limits, execution, and a validation step that decides whether any of it is real. Each link is written down and fixed before money moves.
A model given an account replaces that chain with an opinion. It optimises no stable objective, follows no fixed rules, and has no information advantage: it reads the same public prices and headlines as everyone else. In the following season, on US stocks, Bloomberg counted 6 profitable results out of 32, and two models given the same prompt traded 158 times and 1,418 times. That is not two strategies. It is one instruction producing two unrelated behaviours.
The organisers flagged a further weakness themselves: the models were fed raw numerical time series, a format language models handle poorly.
3. The model belongs upstream of the decision, not in it
None of this means LLMs are useless in trading. It means they sit in the wrong seat. The useful work happens before the decision: writing and reviewing backtest code, summarising papers, turning a vague idea into a testable hypothesis, keeping a record of what was tried. That is where the time savings are real.
The decision itself stays a rule you can test. Man Group, which runs an agent that proposes trading signals, holds AI-generated ideas to the same thresholds as human ones. The model can suggest the hypothesis. It cannot be the evidence.
There is a limit to this argument. A sentiment score produced by a language model can be a perfectly good feature inside a tested strategy, provided the test runs after the model’s training cutoff (more on that next week). The point is not that the model has no use. It is that it does not replace the strategy.
A practical rule: to tell a true Sharpe of 1 from zero with 95% confidence, you need roughly six years of returns. Anything shorter is a demo.
Next time you see an AI trading result, ask two questions. How long was the test, and who wrote the decision rules? If the answers are “two weeks” and “the model”, you are looking at a coin flip with a nice interface.
Over the next few issues, I will share some pieces of how I use these models in my own research. Not all of it: the complete process, step by step, lives in AI QuantPod.
→ See what’s inside AI QuantPod.