Working note No. 9 — on ideas that are real but not good enough.
Abstract. Between the graveyard and the reference configuration there is a third place, and this note is about it. Several ideas in our recent research cycle produced effects we judge to be genuine — statistically present over long samples, mechanically sensible, occasionally elegant — and were nonetheless not adopted, because they failed a recency test, an effect-size bar, or a resampling requirement that we enforce precisely so that "real" and "shipped" remain different words. We describe the drawer, defend its existence against both of its critics, and argue that the adoption bar — not the discovery — is where a research program's character is actually decided.
1. Introduction
Public discourse about quantitative research admits two endings: the idea worked, or the idea failed. Our experience requires a third. Some ideas are, on the evidence, real — the effect exists, the folds agree it exists, the mechanism survives scrutiny — and still do not enter the reference configuration. They go to the drawer: documented, dated, revisitable, and off.
The drawer is not a euphemism for the graveyard. Graveyard residents were refuted. Drawer residents were measured and found insufficient — a different verdict, reached for different reasons, with a different revisit policy.
2. The gate, stated once
Adoption is a conjunction, and we find it clarifying to write it as one. A candidate modification ships only if
procedure ADOPTION-GATE(candidate, base):
d* ← perf(candidate) − perf(base) # improvement on the realized path
for b in 1..B: # resampled histories
path_b ← block-resample(joint history)
d_b ← perf(candidate | path_b) − perf(base | path_b)
ship ⇔ d* ≥ τ # effect-size bar, after costs
∧ Pr_b( d_b > 0 ) ≥ q # robustness across paths
∧ recent-regime check passes # the sample we actually trade
(τ, q, and the recency window are fixed before the study runs)
Nothing in the gate asks whether the effect is real; that question was settled earlier, by the validation protocol of note No. 4. The gate asks whether the effect is sufficient, which is a different interrogation with different failure modes — as the residents below illustrate.1
3. Three residents
The long-memory variant. A conditioning scheme built on a much longer lookback tested as real over a decade of history — comfortably so — and then failed the most recent regime. A decade of significance is exactly the kind of evidence that gets ideas shipped, and exactly the kind that seduces: the recent failure is one draw, but it is the draw we would have been living with. It waits, tagged for a second study when the current regime has written more history.
The diversifier book. A companion construction improved nearly every point-risk metric we track — drawdown, concentration, the shape of bad weeks. It failed the resampling clause. Fig. 1 shows the anatomy in stylized form: the realized improvement is genuinely there (the red marker), but it sits in the upper tail of its own resample distribution, whose mass straddles zero. An improvement that appears in the realized path but dissolves across resampled paths is, on our reading, a description of one history rather than a property of the strategy.2
Fig. 1. This figure shows the distribution of the candidate book's improvement over the base book across B block-resampled histories (gray bars), together with the improvement on the single realized path (red dashed marker). The realized draw lies in the distribution's upper tail while the distribution itself straddles zero: the improvement is a property of the path, not of the strategy. All quantities are synthetic and units are intentionally omitted.
The crumbs. One study found an effect we are confident is genuine market physics — persistent, mechanically explicable, present across specifications. Its magnitude, after realistic costs, is measured in basis-point crumbs: d* real, d* ≪ τ. We record it with the affection one has for a true small thing, and we do not adopt it. Real is a necessary condition. It was never the sufficient one.
We note in passing a fourth category of non-adoption: overlays that, however motivated — including some drawn from literature we admire — simply never triggered out of sample.3 An intervention that never fires cannot be validated, only carried as complexity. We do not carry complexity that has never earned its place.
4. In defense of the bar
The drawer has two critics. The aggressive critic says: the effect is real, the folds passed it, declining to adopt it is pure timidity. The skeptical critic says: if it did not ship, calling it "real" is self-flattery — the drawer is a trophy room for almosts.
Both critics make the same error from opposite directions: they collapse evidence of existence and sufficiency for adoption into one judgment. We keep them separate on purpose. Existence is judged by the validation protocol. Adoption is judged by the gate above, with its constants set where they are — because a reference configuration is not a museum of true facts; it is a machine whose every component must pay rent in the only sample no search can touch: data that has not yet arrived.
The constants are written down, like the kill criteria, before the studies run. This is the entire defense. A bar adjusted after seeing the result is not a bar; it is a negotiation with oneself, and we know who wins those. The gate is, in Duke's (2018) vocabulary, an anti-resulting device: it prevents one lucky realized path — and a realized path is always one draw — from grading a decision the resamples have already graded differently.
5. Conclusion
Show us a research program's discoveries and we will learn what it is curious about. Show us its drawer — the true things it declined to adopt — and we will learn whether its standards can be trusted. The drawer is where a research program proves that its standard is a standard, by paying the only price that certifies one: leaving real effects, lovingly documented, unshipped.
Notes
- The block-resampling scheme preserves the dependence structure that matters at our horizon; we have verified the qualitative conclusions of this note are unchanged under reasonable variations of block length. B, τ, and q are not disclosed, for the usual reason: an adoption bar is only informative about its owner if the owner cannot be reverse-engineered from it.
- The realized path retains one vote: the construction is scheduled for restudy as new history accrues, under the same constants. The drawer, unlike the graveyard, has a calendar.
- We include in this category timing overlays motivated by the overnight-intraday persistence literature — the same literature whose style this series borrows. The homage is sincere; the folds were unmoved. No configuration we admitted under our ex-ante rules produced a single qualifying event out of sample.
References
Duke, A., 2018. Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts. Portfolio/Penguin, New York.
Harvey, C.R., Liu, Y., Zhu, H., 2016. …and the cross-section of expected returns. Review of Financial Studies 29, 5–68.
López de Prado, M., 2018. Advances in Financial Machine Learning. Wiley, Hoboken.
Keywords: adoption criteria, effect size, resampling, regime dependence, research governance.