How Many Backtests Invalidate a Sharpe Ratio? Try 45

Run one thousand backtests on a strategy with literally no edge — no signal, no alpha, nothing but noise — and the best of them will report a Sharpe ratio of 3.26. Not 0.3. Not 1.0. A number most EA vendors would put straight on a sales page. That figure comes from Bailey and López de Prado's work on selection bias, and it is not a statement about bad strategies; it is arithmetic about how many strategies you tried. Every optimizer pass, every symbol variant, every timeframe you quietly discarded is a trial — and the Sharpe the Strategy Tester prints has no idea how many there were.

The Number You Never Reported: How Many Times You Tried

When an EA developer publishes a result, the disclosure is almost always a single number: Sharpe 2.1, profit factor 1.8, max drawdown 14%. What is never disclosed — often not even to the developer's own memory — is the denominator. How many stop-loss multiples, MA lengths, session filters and symbols were tried and abandoned before this one printed a curve worth screenshotting?

That unreported count is not a footnote. It is the single most important input into whether the reported Sharpe means anything. Statisticians call what happens when you run many tests and report only the winner selection bias under multiple testing — the same mechanism behind the replication crisis in medical research. Harvey, Liu and Zhu's 2016 Review of Financial Studies paper argued that given the volume of historical data mining in finance, a newly proposed return factor should clear a t-statistic hurdle of 3.0 rather than the conventional 2.0 before anyone treats it as real — and that most claimed empirical findings in financial economics are likely false precisely because nobody adjusted for the number of tests run.

Retail EA development recreates that dynamic at higher velocity and with fewer guardrails. An academic at least publishes the specification searched; an optimizer run leaves no trace of the thousands of losers it stepped over.

Forty-Five Configurations Buys You a Sharpe of 1

The numbers here are smaller and more alarming than most traders expect. On roughly five years of data, testing more than about 45 independent parameter configurations is close to sufficient to guarantee you will find one with an in-sample Sharpe ratio of 1 whose true out-of-sample Sharpe is exactly zero. Forty-five. That is a single afternoon of manual tweaking, or a fraction of one genetic optimization generation.

Scale it up and the ceiling rises with it. After 1,000 independent backtests, the expected maximum Sharpe ratio across the set is 3.26 — again, on data with no underlying edge whatsoever. The best result in a large search is not a discovery. It is an order statistic.

The more parameter combinations you try, the more certain you are to find one that looks brilliant purely by chance.

Now consider what modern retail tooling does with that threshold. StrategyQuant X advertises a genetic engine that "combin[es] and verif[ies] millions of different entry and exit conditions, order types and price levels" per strategy-generation run. EA Studio and MT5's own optimizer work on the same principle. A tool that blows past 45 trials in the first second of a run, and past 1,000 before you finish reading the progress bar, is not searching for an edge — it is manufacturing an order statistic and attaching a Sharpe ratio to it. As the MQL5 Traders' Blog put it in an August 12, 2026 post on exactly this problem:

A backtest that looks perfect is not evidence the strategy works — very often it is evidence of the exact opposite.

Where the Noise Ceiling Comes From

The 3.26 figure is not a simulation artifact; it falls out of extreme value theory. Draw N independent Sharpe estimates from a distribution centered on zero and the expected maximum is approximated by:

E[max{SR_n}] ≈ E[{SR_n}] + sqrt(V[{SR_n}]) * [ (1 - γ)·Z⁻¹(1 - 1/N)
                                            + γ·Z⁻¹(1 - 1/(N·e)) ]

// γ ≈ 0.5772 (Euler–Mascheroni constant)
// Z⁻¹ = inverse standard normal CDF
// N   = number of independent trials

The practical reading is simpler than the notation. The term that matters is Z⁻¹(1 - 1/N): as trial count N climbs, the inverse normal CDF pushes the expected best result further into the tail. The baseline noise ceiling rises mechanically with every additional trial, whether or not any trial contains genuine skill. Your optimizer is not merely searching a space — it is raising the bar your winning result must clear to be distinguishable from luck. Trial count grows in the thousands with a single click; the significance threshold most traders apply grows not at all.

The Deflated Sharpe Ratio: Correcting the Number You Trust

The formal correction is the Deflated Sharpe Ratio, defined by Bailey and López de Prado in their 2014 paper The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality (SSRN 2460551). In the authors' framing, DSR "corrects for two leading sources of performance inflation: selection bias under multiple testing and non-normally distributed returns."

The estimator returns a probability, not a ratio — the probability that the observed Sharpe reflects real skill rather than the best draw from your search:

DSR = Z[ (SR − SR₀)·sqrt(n − 1)
         / sqrt(1 + ½·SR² − γ₃·SR + ((γ₄ − 3)/4)·SR²) ]

// SR₀ = expected maximum SR given the number of trials run
// n   = track-record length (observations)
// γ₃  = skewness of the returns series
// γ₄  = kurtosis of the returns series

Three terms do the work, and each maps onto a failure mode retail backtests share:

• SR₀ — the trial-adjusted benchmark. You are no longer testing your Sharpe against zero. You are testing it against the noise ceiling your own search created. Run 1,000 trials and SR₀ is not 0, it is 3.26.
• n — sample length. A short track record inflates the standard error of the Sharpe estimate. Two years of M15 data feels like a lot of bars; as a sample of independent return observations it is thin.
• γ₃ and γ₄ — skewness and kurtosis. The plain Sharpe assumes normal returns, so strategies with negative skew and fat tails — martingale grids, wide-stop mean reversion, anything that wins small and often — are systematically flattered by it. The DSR denominator penalizes exactly that shape.

You do not need to implement the estimator to use the intuition. A Sharpe of 2.0 from a single pre-specified configuration tested across a decade is a different object from a Sharpe of 2.0 pulled from a 10,000-pass genetic search on eighteen months of data. The Strategy Tester prints the same number for both. Only one is evidence.

Minimum Track Record Length: The Sample Nobody Has

Invert the DSR and you get a brutally clarifying question: how long a track record would you actually need to confirm this Sharpe ratio sits above a threshold at 95% confidence? That quantity is the Minimum Track Record Length, and the answers are humbling.

In one worked example from portfoliooptimizer.io, a strategy required 1,404 monthly observations — roughly 117 years — to statistically confirm its Sharpe exceeded the threshold at 95% confidence. In the same analysis, a 179-month out-of-sample record, about fifteen years, was judged too short to settle the question.

Fifteen years of genuine out-of-sample performance is more than almost any retail EA will accumulate, and it was still insufficient. Set that against a typical MT5 validation: two to five years of history, most of it consumed by in-sample optimization, with a hold-out measured in months. For most retail configurations the sample simply cannot carry the claim being made from it.

This is where independent data verification earns its keep. Before trusting a Sharpe ratio, confirm the trade count and the true out-of-sample window against a price source outside your broker's own history. Traders can cross-check these windows using TradingView, which provides long multi-decade historical data and bar-replay tooling for building a genuinely untouched hold-out period independent of the MT4/MT5 server's tick archive.

PBO: The Test That Doesn't Require the Formula

If the DSR's moment estimators feel impractical for a retail workflow, the companion framework is more tractable. The Probability of Backtest Overfitting (Bailey, Borwein, López de Prado and Zhu, 2015, Journal of Computational Finance) uses combinatorially symmetric cross-validation (CSCV) and asks one question that requires no distributional assumptions:

When I pick the best configuration in-sample, how often does it rank below the median out-of-sample?

The procedure is mechanical:

1. Split the return series into an even number of equal-length blocks.
2. Enumerate every way of splitting those blocks into a training half and a testing half.
3. For each split, identify the configuration with the highest Sharpe in the training half.
4. Record where that same configuration ranks among all configurations in the testing half.
5. PBO is the fraction of splits where the in-sample winner landed below the out-of-sample median.

A PBO near 0.5 means your selection process has no predictive value — picking the in-sample best is a coin flip against the out-of-sample median. Crucially, this runs on optimization results you already have — every configuration your optimizer evaluated is an input, so the trial count that was destroying your Sharpe estimate becomes the raw material for measuring the damage.

Auditing Your Own Optimizer Runs

None of this makes backtesting worthless. It makes the trial count a mandatory disclosure — first to yourself. Five practices convert the theory into workflow:

1. Log N before you look at the winner. Record the total number of configurations evaluated across every pass, including abandoned symbols and timeframes. If you cannot state N, you cannot interpret the Sharpe.
2. Raise the hurdle, not the search. Harvey, Liu and Zhu's t-stat of 3.0 rather than 2.0 is the right instinct. Treat additional trials as something that raises your required threshold, not something that improves your odds.
3. Keep a genuine one-touch hold-out. A period the optimizer never saw, evaluated exactly once. The moment you re-optimize after seeing hold-out results, that period is consumed and its N merges with the rest.
4. Use purged and embargoed cross-validation. Standard k-fold leaks information across adjacent folds in serially correlated data. Purging overlapping observations and embargoing the bars immediately after each test fold closes that leak.
5. Demand an economic rationale. If you cannot articulate why an edge should persist — what structural behavior or participant constraint produces it — you have a curve-fit, and no statistic will rescue it.

Practitioners also flag red flags that should trigger scrutiny rather than celebration: Sharpe ratios above 5, profit factors above 4, win rates above 90% across hundreds of trades, near-drawdown-free equity curves, and — most diagnostic of all — performance that collapses under small parameter perturbations. A configuration that works at exactly 14 and breaks at 13 and 15 has located a feature of your price history, not of the market.

The reframe is the whole point. "My EA has a Sharpe of 2.4" is not a claim about a strategy. It is a claim about a strategy and a search, and without the second half it cannot be evaluated. Traders who report their trial count alongside their Sharpe ratio may find the number less impressive — and considerably more likely to survive contact with a live account.