Finding the bug that makes your model look good
If a model gets to see data that didn't exist yet, it scores brilliantly in testing and fails the moment it's real. Nothing crashes. No number looks strange.
leakprobe works like this. You hand it the function that builds your features, the tables it reads from, and one line per feature naming which tables that feature is allowed to read. It then makes a change your code should be blind to — pushing one table's timestamps three weeks later, so rows that were visible no longer are — and runs your function again.
A feature that genuinely cannot read that table gets identical inputs through identical code and returns bit-identical numbers: its difference is 0.0, not 1e-15. So any movement at all is proof of a dependency nobody declared. You never need to know what the correct value was, which is the point — if you knew that, you wouldn't have the bug.
- What I proposed
- Test a feature pipeline for data leakage without knowing the right answers: declare which tables each feature may read, change something the code must be blind to, and treat any movement as proof of an undeclared dependency. Behind it, bias fingerprinting: each data-handling mistake in a stock-picking study leaves a fixed pattern of shifts across 22 standard signals, so a published results table can be read backwards to say which mistake it contains.
- What it produced
- The test found a real leak that had inflated a stock-picking result by 59%. Released as pip install leakprobe; on five public datasets it catches 8 of 9 planted leaks with no false alarms, and the one it misses is invisible to it by construction. The fingerprint paper shows a dating mistake inflates and a survivorship mistake rearranges, and that only the second is identifiable from outside.
Nothing has moved yet. Drag the clock.
Move the clock on one data source. Anything that shifts had a dependency on it that nobody declared.
The trick
Some of your features can't possibly depend on when a particular table arrived. A feature built only from daily prices cannot care what day an earnings report was published. That's not a guess about the code — it's arithmetic.
So: push that table's arrival date a month later and run everything again. The features that genuinely can't see it return identical numbers. Not close — identical, to the last decimal, because they got the same arrays through the same code. Anything that moves has a connection to that table you didn't know about. And in a system that predicts the future, an unknown connection to when data arrived is exactly the bug you're looking for.
You never need to know the correct answer. You only need to know one change the answer must survive.
That last line is a real bug. avg_severity filters support tickets on when they were opened rather than when they were resolved — so tickets still open at scoring time leak in, which are exactly the ones that predict churn. It declares that it reads tickets, then doesn't react when the ticket clock moves, because it reached around the clock entirely.
It caught a real one
In the project this came out of, a signal called turnover_1m was filed under “built from prices and trading volume.” It lives in a file called price.py. Every person who looked at it agreed.
It divides by shares outstanding — a number that comes off a company filing. So it was quietly reading regulatory data while sitting in the group that supposedly couldn't. Nobody found that by reading the code. The test found it, because it was the one member of its group that moved when the filing calendar shifted and eleven others held at exactly zero.
Where the idea came from
This started as a question about stock-picking research, where the same bug has a name: using a company's quarterly figures on the day the quarter ended rather than the day they were actually published. Roughly, betting on a game with tomorrow's newspaper.
I measured what that costs. Take 22 stock-picking rules, run them on identical data once correctly and once with the mistake introduced, and compare. Measured skill went up 59%, and four rules that were worthless crossed the line into statistically significant. But it only touched the eleven rules that read company filings. The other eleven use nothing but prices, and they came back identical to the last decimal.
That block of exact zeros is the whole method. It is what turns “this number changed a bit” into “this feature has a dependency it should not have,” and it is what leakprobe generalises out of finance.
The chart shows a second mistake alongside the first: building your test from today's list of big companies and running it backwards, which quietly fills your sample with firms that were small a decade ago and then grew. It damages a completely different set of rules, in a different direction — which is a finding in its own right, and the subject of the paper below.
Bias Fingerprints
A write-up of the measurements above: what each mistake does to 22 stock-picking rules, why one of them doesn't inflate results at all but rearranges which rules look good, and a method for reading the damage backwards to identify which mistake a study contains. The method's validation protocol has been run: the universe mistake is identifiable from a results table alone, 99% of the time in simulation, and the dating mistake is not, which the geometry predicts. The gate statistics were recomputed in October 2026 after a unit mismatch was found; the verdicts did not change, the numbers did, and the paper says so.
The pipeline underneath
None of this works without a pipeline that is correct on the dimension being studied, because the correct run is what everything else is measured against. That is a model ranking about 500 large US companies each month, buying the ones it expects to do well and betting against the rest, built on records of which companies were in the index on any past date, including ones that have since left it, and on government filings dated to the day they were published. One known gap: the price history comes from a free source that drops companies after they delist, so the baseline is itself partly survivorship-filtered. The paper says so.
The five bad years aren't a mistake. The model buys cheap, unglamorous companies, and 2014 to 2018 was the worst stretch for that approach in modern history.
1.9% a year is not a finding, and it is not meant to be one. It is the control condition: the number you get when nothing is wrong, which is what makes it possible to price each mistake against it.
Two things worth knowing about any backtest
Building the instrument produced two results that have nothing to do with this particular model. The first: most reported strategy returns are partly just exposure to a rising market, which an index fund gives you for free. Here 59% of the apparent edge turns out to be exactly that.
The second: machine-learning models contain randomness. Running the identical model six times, changing only that internal randomness, moves the headline score by a factor of five while the underlying signal barely moves. A single reported score is a draw from that spread, not a measurement of it.
How the pipeline got built
The pipeline began as an inherited version reporting 35% a year. Reproducing it turned up six separate problems, none of which raised an error or produced an implausible number — which is what made them worth studying, and what eventually turned into leakprobe. Every one produced output a reviewer would accept.