The schema is not the cost
Ask a model to pull ten things out of a document instead of one, and it finds a smaller share of them. Unless you make it fill in a form, in which case it doesn't.
The problem
When you ask a language model for structured output — a list of records, not a paragraph — you usually hand it a schema and force the answer to fit. The schema guarantees you get valid JSON back.
It also blocks the model from saying what it wanted to say. That has a known cost: several percentage points of accuracy across open models, and at least one published case where letting the model write freely and parsing afterwards beat the schema outright. So the folklore is that grammars make models dumber.
I had a specific reason to check. In my extraction work I'd written down constrained decoding as the most likely alternative explanation for something I was seeing, then never tested it — which is a comfortable way to stay wrong.
What I did
The hard part is knowing the right answer. If you take real documents and have someone mark up every company mentioned, you get labels that are expensive and unreliable — and an annotator's attention drifts on exactly the long lists where the effect should show up.
So I built the documents instead. Take sentences from real SEC filings, each one naming exactly one company, each verified to appear word-for-word in the source. Drop k of them at random positions into filler text drawn from the same filings with every company name stripped out. Pad everything to the same length.
Now the right answer is whatever I put in. No annotation, and k is exact.
0.5 items emitted · precision 0.917
0.8 items emitted · precision 0.882
At k = 1 free generation is ahead by 20.0 points. The schema is costing recall here, which is what everyone expects it to do.
Every number is measured. The three stops are the three levels actually run, 75 documents each, with nothing interpolated between them.
What I found
Recall does fall as the list grows — but only when the model is writing freely. Under a schema, on the same documents, it doesn't fall at all.
The test was fixed in advance: to count, recall had to drop at least 0.15 from the short lists to the long ones, and clear p < 0.05. Free generation drops 0.214 at p = 0.011. The schema drops nothing — recall actually rises, p = 0.73.
The reason is visible in how many items each one emits. At sixteen available, the schema produces 9.4 and free generation produces 7.2, at the same precision. Left to decide when to stop, the model stops early, and stops earlier the longer the list gets. A schema takes that decision away, because a half-filled array isn't valid JSON, so it keeps going.
The part that took longest
Before measuring anything I set a sanity check: with one item hidden in one document, the extractor should find it at least 80% of the time. It came back 33%. Then 40%. Then 40% again, after fixing four separate bugs I could see by reading the output.
Guessing wasn't working, so I measured instead — same sentence, same single item, only the amount of surrounding filler changing.
I had built the filler by picking text for having no company names in it, which made it far emptier than any real document. The sanity check was never failing because of the bugs I kept fixing. It was failing because I had buried one sentence in a haystack nothing like the ones the model actually sees.
That also surfaced a constraint worth stating. The number of items, how densely they're packed, and how long the document is are not three independent knobs — fix any two and the third follows. So every study of this kind picks two and inherits a bias in the third, and which two you pick decides whether your answer is an over- or under-estimate.
Why it matters
If you run structured extraction in production, the schema is not what's costing you recall on long lists. It's what's holding recall up. Anyone considering free-form generation with post-hoc parsing to escape grammar masking should measure this axis first — on this model the trade goes the wrong way, and it costs nothing in tokens to find out.
And the thing I'd keep: I had the alternative explanation written down, in my own notes, marked as untested. Writing down what you haven't ruled out is not the same as ruling it out. The gap between those two was the entire finding.