A new paper, written with Zivana Zerjal. Every defect sat where two paths that should agree had quietly drifted apart. Free PDF below; submitted to SSRN.
The first test a backtest has to pass is not a statistical one. It is this: take a day the live system actually traded, replay it through the backtest, and check that the two made the same decision at the same price and size. If they didn't, nothing the backtest says afterwards is worth much.
We ran that test on one intraday US equity system and kept running it. Over a few months it caught six defects. What unnerved us is that not one of them showed in the summary statistics. Returns looked plausible. Hit rates looked plausible. The numbers were wrong in ways that only a narrow, concrete look could see — one trade against the cache, one bar from four sources, one day's decision record re-read.
The paper writes each one up as symptom, magnitude, root cause and the test that now guards it. Here is the short version.
1. The default data feed was not the consolidated tape
We asked the vendor for historical bars without pinning which feed. It returned single-exchange data. Opening-minute volume in our cache was 0.1–2% of the real figure, on every single trading day we checked, while full-day totals were correct — so nothing looked off. Every feature built from opening-minute data was wrong. One month replayed both ways understated, the next overstated; one losing trade became a winning one. A permutation test we had already run was voided.
Lesson: pin every vendor parameter that changes what the data means, and then check, at the point of use, a property the bug would break.
2. Dividend-adjusted history against unadjusted live fills
A commit switched the fetch to dividend-adjusted prices. That restates every historical bar on today's price basis, which is fine for charts and wrong for execution: live orders fill at the unadjusted price. 37 of 98 symbols shifted, every one a dividend payer. One replayed trade combined an unadjusted entry with an adjusted exit and came out far from the live result. Two full runs were thrown away.
Lesson: adjusted prices answer "what was this worth in today's terms". A backtest must answer "what would my order have filled at that day".
3. A sizing step that scaled past the binding limit
Sizing is a chain of steps. The strategy applied the account's buying-power limit correctly; a later step, one that can increase size, multiplied the already-limited notional with no second check. 20 of 61 simulated fills were sized at a notional the live account can never take — about 18% too large. Live was never exposed, because it re-checks real buying power at order time. The backtest had a field live doesn't.
Lesson: apply the binding limit last, or re-apply it after every step that can increase size. Test with a position already at the limit — that is the case that breaks.
4. The clock is UTC. The market is New York.
A research scan picked the opening bar with the fixed key "13:30". That is 9:30 New York only in summer. In winter it is 8:30, an hour before the open. Thirty percent of the study period was winter; on those days 16% of stock-mornings were scored on a pre-market bar and the rest were silently dropped by a completeness check. Eleven scans were affected. No verdict flipped. Almost every number moved. A reviewer who checks only verdicts would never see it. The same trap had been logged months earlier as an operational note, not turned into shared code — so it came back.
Lesson: never compare clock times as strings. Convert once, in one shared helper, and put a winter day and a summer day in every time-dependent test.
5. Data the live path downloads and the replay did not
One real-money day would not reconcile. Three weeks later we chased it: the live system downloads data beyond the stocks it trades, and one decision input depends on it. The replay built its request from the list of traded stocks. The two lists had drifted apart. With that data missing, the input silently read zero in every harness run for about four months. It feeds several decision rules. Restoring it changed one real-money decision in twenty — and re-explained a live-versus-replay gap we had wrongly blamed on quote timing.
Lesson: build the replay's data request from the live system's own list of required inputs, never from the list of things it trades. Never let a missing input fall back silently to a neutral value. Chase every unexplained parity divergence to the end.
6. "Enabled in config" is not "connected in the execution path"
A family of three. An exit rule that was switched on and ignored, because its reason code was not in the honored set — and the simulation stopped at the ignored exit, hiding every later one. A sizing rule wired only into a backtest-only branch; two live trades were sized as if it didn't exist, and its "end-to-end confirmation" had checked that the flag loaded, not that orders changed. And 53 settings read by production code but never declared — a quarter of the surface — including one that turns order execution on, living in a file nobody audited.
Lesson: for every setting, test that it changes something observable in the path that places orders. Then check behavior in production logs, because the config file may not be the only place settings live.
What the six have in common
Each sat between two paths that should have agreed: a historical request and a same-day one, a field that exists only in the backtest, a data list built from what is traded rather than what is required, configuration and the code that actually places orders. In five of the six, the path closer to live was right.
None was found by reviewing results. One erred in both directions. One moved almost every number and flipped no verdict. One changed a single decision in twenty. One produced a false pass because two errors cancelled. A bug that makes results absurd gets caught. These made results look plausible.
And written lessons did not stop recurrence. The clock trap came back months after it was logged. The sizing-chain lesson came back in a step its own test did not cover. Tests are necessary. They are not sufficient either — one class recurred past its test. The honest state of the paper's own evidence: every figure produced while Defect 5 was present carries it, and none has been re-run yet.
The checklist
- Pin the data feed in every request; test the parameter.
- At decision time, alarm if opening volume collapses across the whole universe at once.
- Use unadjusted prices for execution simulation; test the parameter.
- Replay live trades regularly; after any data change, match one fill to the cent.
- Re-apply the binding size limit after every step that can increase size; test at the limit.
- Convert timestamps in one shared exchange-time helper, tested on a winter and a summer day.
- Build the replay's data request from the live path's required-input list; make missing inputs loud.
- Chase every unexplained replay-versus-live difference to its root cause.
- For every setting, test that it changes an observable action in the order path; declare every key the code reads.
- Check live behavior in logs daily, not settings.
Where to read it
The full paper — eight pages, all six defects with magnitudes, the parity evidence, a worked example of the request-recording test, and an unusually long limitations section — is on the papers page as a PDF. It has been submitted to SSRN and is under review; the SSRN link and DOI will be added when it posts.
One sentence from the abstract that we mean exactly as written: we make no claim about whether the system is profitable. The paper is about whether a backtest is even measuring the system it claims to measure. That question comes first, and it is the one almost nobody tests.
Keywords: backtesting, backtest–live parity, implementation error, research infrastructure.