Working note No. 4 — combinatorial cross-validation as editorial policy.
Abstract. We describe the validation protocol that governs what enters our reference configuration: combinatorially purged cross-validation with kill criteria fixed before any test is run. The protocol's defining property is that it says no almost always, including to ideas we were fond of, and on several occasions to ideas that had already survived every simpler test we could design. We argue that the correct mental model for out-of-sample validation is not a measurement instrument but a journal with a hostile referee — and that a research program which cannot precommit to accepting rejections does not have a validation protocol, only a ratification ceremony.
1. Introduction
The failure mode of practitioner backtesting is rarely fraud and usually seduction: a sequence of small, individually defensible decisions — one more parameter nudge, one more filtered regime, one more "sanity re-run" — each made after seeing the data. The literature has named the endpoint of this road. Bailey and López de Prado's probability of backtest overfitting (PBO) formalizes the intuition that a strategy selected as the best of many trials is, with computable probability, the luckiest rather than the best; their companion deflated Sharpe ratio restates a backtested Sharpe after charging it for the number of trials, the non-normality of returns, and the length of the sample.1
Our response was institutional rather than statistical: we adopted a protocol whose architecture makes the seductive path expensive.
2. The protocol, in outline
Three components matter; none is original to us, and all are documented in the public literature, principally López de Prado (2018), where the machinery below — CPCV, purging, embargoes — is developed in full.
Combinatorial folds. Rather than a single train-test split — one draw from the space of histories — the sample is partitioned into groups and every combinatorial assignment of training and testing groups is evaluated. Fig. 1 shows the scheme: each row is one split, each strategy faces every row, and a candidate must be profitable not on a path but across a distribution of paths. Fragile ideas, whose performance lives in one regime, reveal themselves as exactly that.
Fig. 1. This figure shows the structure of combinatorially purged cross-validation for an illustrative configuration of six temporal groups (columns G1–G6) and the resulting splits (rows S1–S6, a subset shown). In each split, dark cells are test groups, light cells are training groups, and the thin red bands mark purged boundaries — observations adjacent to a train-test frontier whose labels overlap in time are removed from training. Our working configuration differs in its group count and split enumeration; the figure is schematic.
Purging. Observations adjacent to fold boundaries, whose labels overlap in time, are removed, so that information cannot leak across the train-test frontier through the mundane channel of overlapping horizons. Leakage of this kind is, in our experience, the single most common manufacturer of false confidence in short-horizon research.2
Ex-ante kill criteria. Before the first fold runs, the acceptance region is written down: the fraction of folds that must be profitable, the significance threshold — set with the Harvey-Liu-Zhu (2016) argument in view, which is to say materially above the textbook hurdle — and the PBO we will tolerate. After the folds run, there is nothing to discuss. The criteria were chosen by people who did not yet know the answer; they are, in that precise sense, the only disinterested parties in the room.
3. Living under the protocol
Two consequences of this regime deserve notice.
First, it kills things one loves. A conditioning idea with a coherent economic story, months of encouraging exploratory results, and — we admit — internal advocates, went to the folds and returned with a unanimous rejection: profitable in none of them. The story had been good. The folds did not care. We record, as an empirical matter, that the disappointment fades in roughly a week and the months it would have burned do not come back at all.3
Second, and symmetrically: what the protocol passes, it passes with an authority no enthusiasm can manufacture. The few ideas that have cleared it did so with fold-level consistency and overfitting probabilities that left little room for the lucky-draw hypothesis — and they entered the reference configuration carrying a confidence that survived their first out-of-sample drawdown, which is the only moment confidence is worth anything.
4. Conclusion
A validation protocol is a constitution: its value derives entirely from the fact that it binds when one wishes it would not. Ours has rejected, at this writing, nearly everything ever submitted to it — including, on the record above, most of our own judgment. We regard this as the protocol working.
The referee is hostile. That is what we pay him for.
Notes
- Bailey and López de Prado (2014) supply the closed form; the operational point is that a Sharpe ratio unaccompanied by its trial count is not a statistic, it is an advertisement. We compute the deflated version as a matter of course and treat any candidate whose significance evaporates under deflation as never having been significant.
- Purging is complemented by an embargo: a buffer following each test group excluded from training even where labels do not formally overlap, as insurance against serial dependence the labeling scheme does not capture. Both practices follow López de Prado (2018, ch. 7).
- Duke (2018) would classify the residual temptation — "the folds must have missed something, the story was so good" — as textbook resulting in reverse: grading a process by the outcome one wished it had produced. The protocol's function is precisely to make that grading impotent.
References
Bailey, D.H., López de Prado, M., 2014. The deflated Sharpe ratio: correcting for selection bias, backtest overfitting, and non-normality. Journal of Portfolio Management 40 (5), 94–107.
Bailey, D.H., Borwein, J.M., López de Prado, M., Zhu, Q.J., 2014. Pseudo-mathematics and financial charlatanism: the effects of backtest overfitting on out-of-sample performance. Notices of the American Mathematical Society 61, 458–471.
Duke, A., 2018. Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts. Portfolio/Penguin, New York.
Harvey, C.R., Liu, Y., Zhu, H., 2016. …and the cross-section of expected returns. Review of Financial Studies 29, 5–68.
López de Prado, M., 2018. Advances in Financial Machine Learning. Wiley, Hoboken.
Keywords: combinatorial cross-validation, purging, probability of backtest overfitting, deflated Sharpe ratio, kill criteria.