The lab's validation machinery, written up properly and posted on SSRN.
For two months this site has published working notes about ideas that failed our tests. A fair question kept coming back: what exactly is the test? The answer is now a 13-page paper, free on SSRN, with every number reproducible from public simulation code.
What's inside
A Monte Carlo of both error types. We simulated thousands of strategy families with no real edge, then measured how often common research habits approve them anyway. Tuning on the full sample: 36.9% false approval. Selecting on a reused holdout: the same 36.9% — a closed-form identity, because a holdout you consult for selection is not a holdout anymore. A disciplined single holdout: 2.3%. A hurdle that charges for the number of trials behind a result: 0.4%.
An uncomfortable finding we published anyway. The popular "profitable in most folds" heuristic passed 88% of pure noise in our experiments. Selection contaminates every fold, so the rule audits almost nothing. We'd been treating fold counts with more respect than they earned.
A new criterion: leave-best-era-out. Drop the candidate's best-performing era and demand it clear the hurdle on what remains. Era-locked candidates — real effects confined to one regime — go from fooling 92% of the standard stack to passing 1.2% of the time. The cost is measured, not hidden: detection of modest stable effects falls too, and the paper reports exactly how much.
The burial-record system. The template this lab uses to turn every rejection into a durable, searchable record — including an operational rule for the one quantity every deflated statistic depends on and almost no presentation reports: how many trials were actually run.
Where to get it
The paper is open access under CC BY-NC-ND: ssrn.com/abstract=7260379. The simulation code is a single Python file with a fixed seed — it reproduces every table in a few minutes on a laptop. If you find an error, that's the process working; tell me and the next revision will say so.
Keywords: backtesting, multiple testing, negative results, cross-validation, pre-registration.