This lab — posted on SSRNSSRN Abstract 7260379 · Aug 2026
Test Everything, Publish Both Results: A Protocol That Cuts Backtest False Positives from 37% to 0.4%
Alex Vidovich · 2026
In brief
Four artifacts for systematic strategy research: a taxonomy of recurring backtest failure modes with testable diagnostic signatures; a Monte Carlo study of both error types across common research behaviors; a pre-registrable era-robustness criterion (leave-best-era-out); and the burial-record template that turns each rejection into durable research infrastructure, including an operational rule for counting trials — the quantity deflation depends on and presentations omit.
Field report
The simulations are fully public and synthetic, so the numbers can be said out loud. Under a pure null with twenty variants per idea, tuning on the full sample falsely accepts 36.9% of candidates — and selecting on a reused holdout is the identical quantity, because a holdout consulted for selection is not a holdout. A disciplined single holdout achieves its nominal 2.3%; a trial-count-deflated hurdle achieves 0.4%. The popular fold-consistency heuristic passes 88% of noise — near-vacuous once selection is accounted for.
Against an era-locked alternative — a real effect confined to one regime — the standard stack fails almost completely (92% acceptance), while leave-best-era-out collapses acceptance to 1.2% at a null size of 0.04%. The price is measured rather than hidden: detection of a stable SR 0.5 on eight years of dailies is 28% under deflation, 7% under LBEO, and falls further when trial variants are realistically correlated. Severity is purchased with power at exactly the effect sizes that survive costs.
Verdict: Posted on SSRN August 14, 2026 · 13 pages · every number reproducible from the public simulation code
Keywords: backtesting, multiple testing, negative results, cross-validation, pre-registration
Journal of Financial EconomicsJFE 134 (2019) 192–213
A Tug of War: Overnight Versus Intraday Expected Returns
Dong Lou, Christopher Polk, Spyros Skouras · 2019
In brief
The authors decompose close-to-close returns into their overnight and intraday components and document a striking division of labor: firm-level return continuation lives in one component, reversal in the other, and the split persists for months — across the cross-section, across strategies, and across nine international markets. Their interpretation is a tug of war between clienteles who prefer to trade at different times of day. Nearly every popular anomaly premium, they show, is earned either entirely overnight or entirely intraday — almost never both.
Field report
This is the paper we have spent the most research hours interrogating, and the field report is two-sided.
As a lens, it is permanently adopted. Decomposing every diagnostic into overnight and intraday components — the paper's basic move — changed how we read our own results: several behaviors that look like noise in close-to-close terms resolve into clean structure once the clock is split. The clientele framing also gives the most convincing answer we know to the question every systematic researcher gets asked: why would an edge survive being published? Because the other side of the trade is not uninformed — it is institutionally committed to trading at a particular time of day, and that commitment does not read papers.
As a timing overlay, it did not convert — for us. Overlays motivated by the paper's smoothed overnight-minus-intraday spread went through our standard protocol and no admissible configuration produced a single qualifying event out of sample (working note No. 9 records the funeral). We stress the scope of that sentence: our horizon, our instruments, our admission rules. The phenomenon in the paper is documented at monthly horizons on broad cross-sections; our failure to convert it says nothing about the paper's claims and a great deal about the distance between a documented premium and an adoptable signal.
Reading advice for practitioners: the introduction, Fig. 2, and Table 2 carry the argument; the international replication is the underrated part — nine markets with the same anatomy is the kind of robustness no backtest of ours will ever have.
Verdict: Adopted as a lens · refuted as an overlay (our horizon) · recommended without reservation
Keywords: overnight returns, intraday returns, anomalies, clienteles, momentum