One Backtest Isn’t Enough (5/8)

Why most strategies fail when the market shifts, and how to avoid it

Aug 01, 2025

In the last newsletter, we structured the strategy: entries, exits, sizing, and risk management.
That was the theory, clean, on paper. Now it’s time to face reality.

Because a well-written strategy, tested naively on past data, means nothing.
The market won’t politely replay the same sequences in the same order, with the same spreads, same fills, same behavior.

A simple backtest, on frozen data, is just a fit check.
It tells you the engine starts. It doesn’t tell you if the car can handle sharp turns in the rain at 80 miles per hour.

The purpose of this phase isn’t to prove your strategy “works.” It’s to try to break it. To see how far it holds.
And most importantly, to understand why it might fail.

That’s the real role of backtesting. Not to chase a pretty equity curve.
But to eliminate false positives, one by one.

In this newsletter, we’ll look at what a real backtest looks like.
Not a naive replay of history. A stress test. And how to go further with robustness checks, realistic simulations, walk-forwards, perturbations, that bring you closer to what your strategy will face in live conditions.

You’re not trying to validate.
You’re trying to survive.

The next step will be about incubation, live trading and monitoring.

👉 If you want to go deeper into each step of the strategy building process, with real-life projects, ready-to-use templates, and 1:1 mentoring, that’s exactly what the Alpha Quant Program is for.

It’s the full roadmap I use to turn ideas into live strategies.

1. Backtest ≠ Replay of the Past

A standard backtest runs your strategy on historical data and shows you how it would have performed.
But if that’s all you’re doing, you’re not testing your strategy.
You’re just applying it to the one version of the past you happen to know.

Here’s the issue:
A strategy can look great simply by fitting a particular sequence of events.
It might benefit from a lucky cycle, a specific volatility regime, or a pattern that just happened to work once.

Let me show you what that means visually:

Each line here represents a plausible simulation of the same asset.
Same starting point. Same statistical structure. But wildly different paths.

If your strategy only works on the one path you happened to backtest on, it’s not robust. That’s not a test. That’s wishful thinking.

Backtesting isn't about validating that something "works." It's about seeing how fragile it really is.

My tip: If your strategy looks great on a single historical path, that’s not proof it’s a red flag. Don’t trust performance until it survives distortion.

2. Try to Break It

Most people run a backtest hoping it works. But that’s not the mindset you want.

You’re not here to confirm your idea. You’re here to destroy it, and see if it survives.

That’s the real goal of backtesting:
You want to identify the exact conditions where your strategy falls apart. Where it underperforms, overtrades, slips, freezes, or does something you wouldn’t tolerate in live trading.

Because if you don’t find those weak spots early, the market will find them for you, and it won’t be gentle.

So instead of asking:
“Does it work on BTC 15-min from 2018 to 2025?”
Ask:

What happens if I start in a different month?
What happens if volatility doubles?
What if slippage increases by 2x?
What if I remove one filter or one asset?

You’re testing boundaries, not scenarios.
The goal is to explore the space around your idea and measure how fast performance breaks down.

A good strategy doesn’t rely on perfect conditions.
It performs well even when you mess with its inputs, its timeframe, its market behavior.

My tip: If your strategy only looks good when everything is just right, it’s not a strategy, it’s a fluke. Robust strategies stay standing when you shake the foundation.

3. Realistic Robustness Testing

Once you've built the habit of trying to break your strategy manually, the next step is to do it systematically. That’s where robustness testing comes in.

These aren’t just technical buzzwords, they’re essential tools to move from fit to reliability.

3.1. Walk-Forward Testing

Split your data into rolling training and testing periods.
Train your model on one slice, test on the next. Repeat.
This mimics how a live system updates over time, and shows whether your strategy can handle evolving market conditions.

If your performance drops hard every time the data shifts, that's a sign your edge is brittle.

Illustration of a Walk-Forward optimization output from the Alpha Quant Program

3.2. Monte Carlo Simulations

Instead of testing a single timeline, generate dozens of alternate versions:

Shuffle the order of trades
Add random delays to execution
Apply noise to prices, slippage, volatility

Why? Because the market won’t replay the past perfectly.
If your strategy only works on a single sequence of events, it probably won't last.

Illustration of a Monte-Carlo simulation output from the Alpha Quant Program

3.3. Time Series Cross-Validation

Most CV methods aren’t built for time.
But with walk-forward CV or nested cross-validation for time series, you can get a better estimate of how your model generalizes.
Especially useful for ML-based strategies where overfitting is harder to spot.

Illustration of a Combinatorial Purged Cross Validation (CPCPV) output from the Alpha Quant Program

Wrapping Up

A backtest isn’t proof that your strategy works, it’s a tool to uncover where and how it might fail.
Robustness doesn’t come from fitting the past. It comes from challenging your assumptions, distorting your inputs, and simulating the unpredictable.

If your idea only survives one path, one setup, one set of parameters, it’s not ready.
But if it can hold through stress, randomness, and change, now you’re onto something.

Next, we’ll dive into how to move beyond testing and actually prepare your strategy for live execution: automation, monitoring, and the traps to avoid once real money is involved.