Test Everything, Publish Both Results: A Protocol That Cuts Backtest False Positives from 37% to 0.4%
- Alex Vidovich · Independent researcher · alexvidovich.com
August 14, 2026 · 13 pages · Working paper · posted on SSRN
Abstract
This paper delivers four artifacts for practitioners of systematic strategy research. First, a taxonomy of recurring backtest failure modes, each with a testable diagnostic signature. Second, a Monte Carlo study of both error types across common research behaviors. Under a pure null with twenty variants per idea, tuning on the full sample falsely accepts 36.9% of candidates (a closed-form result: selecting on a reused holdout is the same quantity, since a holdout consulted for selection is not a holdout), a disciplined single holdout achieves its nominal 2.3%, and a trial-count-deflated hurdle achieves 0.4% — with the popular fold-consistency heuristic shown to be near-vacuous (87% acceptance) because selection contaminates every fold. Under an era-locked alternative — a real in-sample effect confined to one regime — the entire standard stack fails (deflation passes 95%), motivating our third artifact: leave-best-era-out (LBEO), a pre-registrable criterion that collapses era-locked acceptance to 1.2% at a null size of 0.04%, and whose measured cost is power — 7% detection of a stable SR 0.5 on eight years of dailies, versus deflation's 28%, both falling further when trial variants are realistically correlated. Severity, in other words, is purchased with power at exactly the effect sizes that survive costs, and we quantify the price rather than deny it. Fourth, the burial-record template that converts each rejection into durable research infrastructure, including an operational counting rule for the quantity deflation depends on and presentations omit: the number of trials. All results are simulation- and backtest-based; nothing here constitutes investment advice.
Keywords: backtesting, negative results, multiple testing, cross-validation, overfitting, pre-registration, research process
JEL classification: C12, C15, C52, C58, G11, G14