Backtesting guide
Backtest Overfitting: Selection Bias and Validation Design
Backtest overfitting occurs when choosing rules or parameters rewards patterns specific to historical data that do not persist on unseen data. An impressive winning run is weak evidence if hundreds of alternatives were tried. Record the full search, freeze the selection rule, and test on untouched data. Probability of Backtest Overfitting (PBO) measures how often an in-sample winner falls below its alternatives out of sample.
Read the practical guideKey Takeaways
- Count parameter combinations, rule revisions, markets, timeframes, and rejected ideas that influenced selection.
- A test period stops being untouched when its results guide another strategy change.
- PBO needs aligned observations for all candidates; one winning equity curve or summary CSV is insufficient.
- Parameter stability and cost stress tests can expose fragility, but no validation score proves future profitability.
Record the search before judging its winner
Create one trial ID for each evaluated configuration. For example, 5 lookback lengths × 4 entry thresholds × 3 exit settings produce 60 combinations. Running those combinations on 2 timeframes creates 120 evaluated configurations. A later change to the entry rule starts another documented search. This count describes research effort; correlated configurations are not 120 independent statistical experiments.
- Record the strategy version, parameter values and ranges, symbol, timeframe, data dates, costs, sizing, and execution settings.
- Declare the ranking metric and risk limits before examining results. Log every change to that decision rule.
- Keep losing and discarded trials, plus a separate record of failed or incomplete runs. Do not turn a failed run into an invented return series.
- Record discretionary exclusions and manual chart inspections. Degrees of freedom include decisions outside the parameter grid.
Keep selection data separate from confirmation data
An illustrative protocol uses 2019 through 2022 for development, 2023 for validation, and 2024 through 2025 as a sealed final holdout. These dates are examples, not recommended sample sizes. Select rules and a parameter region with development and validation data, then freeze the version, costs, ranking rule, and acceptance criteria before opening the holdout.
- If 2023 results influence a revision, that period is part of development. It cannot also be described as the final independent confirmation.
- At each boundary, define indicator warm-up and the treatment of open positions. Do not let future labels, overlapping outcomes, or future-fitted preprocessing enter selection.
- If the final holdout fails, report that result. Revising the strategy against it consumes the holdout; a fresh confirmation requires new untouched data.
- Choose enough observations for the strategy horizon and market coverage. A calendar split alone does not make returns independent or regimes representative.
What Probability of Backtest Overfitting actually measures
Bailey, Borwein, Lopez de Prado, and Zhu define PBO through the out-of-sample rank of the candidate selected in-sample. Their Combinatorially Symmetric Cross-Validation (CSCV) method divides an aligned observation-by-candidate matrix into an even number of ordered blocks. For each half-block combination, select the best in-sample candidate and rank that same candidate on the complementary blocks.
- With N candidates, rank out-of-sample performance from 1 (worst) to N (best). Set omega = rank / (N + 1) and lambda = ln(omega / (1 - omega)).
- Estimate PBO as the fraction of split logits at or below zero. State the tie policy and median-boundary convention; the example below has no ties or zero logits.
- Keep the candidate set and performance metric fixed. A full-period Sharpe ratio cannot reconstruct the returns needed to recompute a split Sharpe ratio.
- CSCV can place later blocks in training and earlier blocks in testing. It diagnoses selection within the observed matrix; it is not a chronological walk-forward deployment simulation.
Recalculate a six-split PBO example
Use four fictional candidates and four equal-length blocks. The entries below are block mean returns in basis points per observation. Choose the arithmetic mean as the selection metric, with higher values better. Because the blocks have equal observation counts, the score for a two-block set is the average of its two entries. This small example demonstrates ranking mechanics, not a reliable estimate for a real strategy.
- Input matrix, in block order 1, 2, 3, 4: A = [2, 7, 1, 4]; B = [3, -3, 1, 7]; C = [6, 2, 3, 7]; D = [2, 1, 5, -4].
- Train 1+2, test 3+4: winner A scores 4.5 in-sample and 2.5 out-of-sample. Test scores A=2.5, B=4, C=5, D=0.5 give A rank 2 of 4.
- Train 1+3, test 2+4: winner C scores 4.5 and 4.5; out-of-sample rank 3.
- Train 1+4, test 2+3: winner C scores 6.5 and 2.5; out-of-sample rank 2.
- Train 2+3, test 1+4: winner A scores 4 and 3; out-of-sample rank 2.
- Train 2+4, test 1+3: winner A scores 5.5 and 1.5; out-of-sample rank 1.
- Train 3+4, test 1+2: winner C scores 5 and 4; out-of-sample rank 3.
- There are 4 choose 2 = 6 training combinations, including each reversed train/test assignment. The ranks [2, 3, 2, 2, 1, 3] produce logits approximately [-0.405, 0.405, -0.405, -0.405, -1.386, 0.405]. PBO = 4 / 6 = 66.7%.
- All six selected candidates have positive test means, yet four rank below the alternatives' median. PBO is a relative selection diagnostic, not a 66.7% forecast of losing money. The overlapping splits are not six independent experiments.
Check the parameter neighborhood before opening the holdout
Consider an illustrative development sweep of lookbacks 18, 20, and 22 with entry thresholds 1.8, 2.0, and 2.2. At each lookback, list net returns in threshold order: 18 gives [1%, 2%, 1%]; 20 gives [2%, 18%, 1%]; 22 gives [0%, 1%, -1%]. The 20/2.0 setting is an isolated peak. Its eight neighbors have a median return of 1%, compared with its 18%. This is evidence to investigate dependence on exact settings, not proof of overfitting.
- Compare trade count, turnover, drawdown, and exposure across the same grid. A changed trade count can explain a discontinuity.
- Choose the neighborhood and rejection rules before viewing the final test. Expanding the grid until a preferred story appears is another selection decision.
- A broad plateau can still share the same leaked signal or favorable regime. Keep the holdout and execution checks even when neighboring cells look similar.
Stress costs without changing the chosen strategy
Suppose a fictional holdout contains 100 completed round trips with $10,000 entry notional and $10,000 exit notional each, and $3,000 gross profit. A combined commission-and-slippage assumption of 5 basis points per side costs 100 × 2 × $10,000 × 0.0005 = $1,000, leaving $2,000. At 10 basis points per side, costs are $2,000 and net profit is $1,000. At 15 basis points, the $3,000 cost consumes the entire gross profit.
- These are sensitivity calculations with fixed fills, not observed returns. Use actual turnover when notionals vary; futures and options require their own fee and contract conventions.
- Do not subtract baseline costs twice from an already net result. Rerun the engine if changed fills, stop execution, sizing, or compounding can alter the trade path.
- Predeclare cost scenarios and risk limits. Choosing a cheaper scenario after a failure changes the evaluation rather than validating the strategy.
Retain useful evidence with Pineify
Strategy Optimizer supports parameter comparisons and CSV export of stored result rows with parameter configuration details. Preserve those results alongside your research ledger, including counts of unsuccessful attempts. Summary metrics help compare trials, but they are not the aligned return matrix required for PBO. Collect that matrix separately before using an external CSCV implementation.
- Set test windows and freeze the candidate yourself; this workflow does not assume automatic walk-forward testing.
- For a separately prepared holdout, Backtest Deep Report accepts TradingView List of Trades CSV files and analyzes the supplied trades. Keep the strategy version and sample dates with each file.
- A trade report does not establish sample independence or recover discarded research. Neither product is presented here as a PBO calculator.
Direct Pineify workflow
Keep the evidence behind your parameter choice
Compare parameter configurations in Pineify Strategy Optimizer and export the stored results. Keep a separate research ledger and collect aligned returns if you plan to calculate PBO externally.
Review parameter trials and export their stored result rows.
Open Strategy OptimizerAnalyze a TradingView trade CSV from your separately prepared holdout.
Analyze holdout tradesBoundary: Parameter summaries and trade reports do not calculate PBO, make a reused sample independent, or prove future profitability.
Primary sources
This page is educational and does not provide investment advice. Backtests are hypothetical, depend on their data and assumptions, and do not guarantee future results. Trading can result in substantial loss.