Equity factor research
A stock-picking model that claimed to return 35% a year. I rebuilt it from scratch, found six bugs that had manufactured that number, and published the real one instead: 1.9%.
What the project is
A model that ranks about 500 large US companies each month, buys the ones it expects to do well, and bets against the ones it expects to do badly. The aim is to make money whether the market goes up or down.
I inherited a version that reported very strong results. They looked too strong, so before adding anything I tried to reproduce them.
What was wrong
Six separate problems, each producing output that looked completely reasonable. Two of them need no finance background at all:
A third of the calendar was missing
The code traded on the last day of each month. When the 31st fell on a Saturday it found no trading data, skipped that month — and skipped the month's profit and loss with it. 578 of 1,971 days simply weren't there. The results were an average over whichever days happened to survive.
It only knew about companies that survived
The test used today's list of large companies and applied it to the past decade. Every company that went bankrupt or was taken over was invisible — like judging a doctor's record after removing the patients who died.
What I rebuilt
I threw out the data and started again. The new version knows which companies were in the index on any given past date — including ones that no longer exist — and pulls each company's financial statements from government filings, dated to the day they were actually published rather than the day the quarter ended.
That detail matters more than it sounds. A company's Q1 results aren't public until May; using them in April is the most common way to accidentally cheat.
Performance
The five bad years aren't a mistake. The model buys cheap, unglamorous companies, and 2014 to 2018 was the worst stretch for that approach in modern history. A reasonable strategy living through an unreasonable decade.
Skill versus market exposure
This is the finding I care most about. I separated how much of the return came from genuinely picking better companies, versus simply being exposed to a rising market. The answer was uncomfortable.
How stable the result is
Machine-learning models contain randomness. I ran the identical model six times, changing only that internal randomness, to see how much the headline score moved.
What I took from it
The easy version of this project reports 35% and moves on. The useful version finds out the number was an artifact, rebuilds the foundation, and publishes a smaller honest figure with the evidence for why it's smaller.
Telling the difference between a real edge and a measurement error is the job. A firm trading on these signals loses money when someone can't.