Somewhere in your validation code, there is a stack. A HAC correction someone recommended. A deflated Sharpe you added after Lopez de Prado. A cross-validation scheme, because everyone cross-validates. Each layer went in for a good reason, and each one made the whole thing feel a little more serious.
Almost nobody asks what each layer actually changes, in measured false positives. So I did: a validation pipeline is itself a statistical test, strategies in, accept or reject out. I ran it by Monte Carlo on returns where the truth is planted, so I know exactly how often each pipeline gets fooled.
The answer: half the layers change nothing, and the ones that work have a price almost nobody quantifies.
The full study, with planted ground truth and released code, is here
→ https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7210219
1. A test only helps if it covers a hole another test left open
Every control targets one failure mode. Autocorrelation corrections fix the standard error. Multiplicity corrections fix the fact that you screened many strategies. Out-of-sample schemes fix the in-sample search.
The result that matters: two controls targeting the same failure mode are substitutes. Calibrate them to the same false-positive rate and they become indistinguishable. HAC versus block bootstrap: tie. Bonferroni versus Holm: tie. Stacking both does not lower your false-positive rate by a single point. It only destroys power, meaning real strategies thrown away as luck.
The failures themselves are real. An uncorrected test on 20 strategies validates pure noise about 64% of the time, and under strong autocorrelation it runs a 29% false-positive rate instead of the 5% it claims. So you need guards. Exactly one per failure mode present, and nothing more.
2. There is a failure that size and power cannot see
A pipeline can hold its false-positive rate, detect live edges, and still validate a strategy whose edge is already dead. Think of a signal that worked for seven years and died two years ago, quietly, inside your sample. Classical test theory has no name for this, because it assumes the truth does not move.
So I planted edges that break mid-sample, and one scheme stood apart. The strictly forward holdout, which judges only on the end of the sample, treats a dead edge like a null: it validates it 5% of the time, its own error rate. Walk-forward and purged k-fold keep validating it at roughly twice that rate, because they recycle the alive past as out-of-sample. The design that survives every robustness check is a conjunction: a powerful scheme and a forward holdout, and the strategy passes only if both agree.
Against an edge that erodes gradually instead of breaking, no scheme protects: every out-of-sample window still contains some live signal, so every scheme gets fooled equally. That defense lives in monitoring after deployment, not in your validation pipeline.
3. Honesty doubles the edge you need
Now the price. A single honest test on ten years of daily data detects a Sharpe of about 0.8 with 80% power. Screen 20 strategies with 10 configurations each, validate with a pipeline that is correctly sized and forward-honest, and the minimum detectable edge rises to about 1.6.
Twice the edge. Not because the pipeline is badly built, but because it is honestly built. The detection bar scales with the size of your search, and one product governs everything: annualized Sharpe times the square root of years. Read it the other way: the Sharpe 0.8 edge a single test resolves in a decade would take this pipeline four decades to certify.
Reliability is not bought by piling on tests. It is bought by covering each failure mode once, and knowing what your pipeline cannot see.
Next time a validation layer makes you feel safer, ask which failure mode it covers. If you cannot name it, the layer is decoration.