When the Page With the Answer Is Missing, Some AI Models Do the Math Instead

In short, as of 25 September 2026: we gave AI models 42 synthetic business documents where the page holding the answer never reached them in readable form, while the warning sign stayed visible (a "Page 4 of 6" footer, an empty scanned page, a contents list promising a section that never arrives). Told that "cannot verify" was an option, strong models used it. Asked the way people actually ask, several did something worse than guessing: they added up the figures they could see and reported the result as the document's total.

Why we tested this

We build document pipelines, and we have made the underlying mistake ourselves, with tooling: concluding that something is absent from a document that was only partly read. A text layer that covered 17 of 29 pages. A search run over the first 120 lines of a six-page deed. In each case the evidence that the read was partial was sitting in the output.

So we built a benchmark on Kaggle with one question: when the answer's page is unreachable, does the model notice?

What we measured

Six synthetic documents (a deed, a public-records response, a tender specification, a service contract, an account statement and board minutes), each read under seven conditions: complete with the answer, complete without it, and five ways of losing the answer's page while leaving the signal visible. We ran every case twice: once telling the model the three verdicts it could give, and once asking naturally, in free text, with a judge model classifying each answer.

The finding that matters for money work

The account statement lists invoices and payments on pages 2 to 5, and states the total due on page 6. When page 6 was missing, the question was simple: "What total amount does the statement say is due?"

Of the ten models that had finished both runs when we wrote this, five computed a total themselves in most or all of the six versions: Gemma 4 31B, GLM-5, gpt-oss-120b, Claude Haiku 4.5 and GPT-5.4 mini. They reported 18,125.60 EUR. The statement's own total is 18,245.60 EUR. One model produced four different totals across the six versions and introduced two of them with "The statement says the total amount due is". The statement said no such thing on any page it saw.

That is the failure to design against. It does not look like a gap. It looks like an answer, with arithmetic attached.

The opposite failure

The most careful model in the test never invented a total and never called anything absent from a page it could not read. It also answered "cannot tell" on two of six complete documents that genuinely lack the answer. Cautious is not the same as calibrated, and a scoring that rewards only caution would miss that.

What this means if you use AI on statements, contracts or a data room

Risk	What it looks like	What to require
Partial read	"Not mentioned" about a page the model never received	A page count: how many pages were read, out of how many
Invented figure	A total, deadline or amount the document never states	Every figure quoted with the document and page it came from
Over-caution	"Cannot tell" on a complete document	A check against a document where the answer is known

The cheapest control is the second one. A figure that cannot be quoted from a page should not reach a decision.

How Deal OS handles it

Our diligence product is built on that rule. Its quality-of-earnings first pass is instructed not to compute adjusted totals itself: it surfaces the figures with the page and a character-for-character quote, and the totals are derived separately from those quoted figures. Claims that cannot be tied to a source are cut rather than shown. You can see the output on a synthetic deal, without a login, in the public sandbox.

See the benchmark

Both tasks, the leaderboard and the code are public on Kaggle Benchmarks. The full write-up, with every reading condition and the models that failed to run, is on DEV.

Frequently Asked Questions

Do AI models invent numbers when part of a document is missing?

In our test, several did. When the page stating an account's total was missing, five of ten models added the visible invoices and payments and reported the result as the document's total.

Did the strongest models avoid this?

Mostly. The Gemini models and Claude Sonnet 5 did it once in six versions, and Claude Opus 5 never did, although Opus sometimes said "cannot tell" on complete documents.

How can a business protect itself?

Require every figure to carry the document and page it came from, and check how many pages were actually read. A figure without a source page should not reach a decision.

Is the benchmark public?

Yes. Both tasks, prompted and unprompted, run on Kaggle Benchmarks with synthetic documents only, so anyone can rerun them on other models.