Resampled Thought Trees

Modern AI systems solve hard problems by exploring — trying one line of reasoning, then another, like a person working through a puzzle. At each step the model picks which idea to pursue next, based on how promising it looks.

The catch: nobody knew whether “looks promising” and “actually leads somewhere” are the same thing. Everyone measured whether the final answer was right, which tells you nothing about which steps along the way were doing the work.

Freeze the AI at one specific thought. Restart it from exactly that point, ten times over. Count how often it still reaches the right answer.

Do that for every thought in the tree and you get a score for each one: how much that step was really worth. We ran it 22,350 times across two kinds of puzzle.

The AI's own confidence is a poor guide. Its top-ranked next step was frequently a dead end, while options it ranked last turned out to be the ones that reliably worked.

That matters practically: these systems decide what to explore using exactly that ranking. If the ranking is unreliable, they spend their budget in the wrong places.

We expected a simple rule like “early steps matter most.” Instead the two tasks behaved in opposite ways — so there is no shortcut, and you have to measure.

We also deleted branches from the tree to see what broke. Judged only on did it still solve the puzzle, deleting things looked harmless.

But counting how many different routes to the answer survived told another story: some branches quietly removed most of the solutions while leaving one behind. The puzzle still solved, so a normal evaluation saw nothing — while the system had become far more fragile.

This was a five-author project at USC. The honest framing of the results: we controlled the AI's options rather than letting it generate freely, which makes the measurement much cleaner but means the findings describe this setup rather than every AI system.