Why a Scan Is Not a Document

In the previous piece I argued that a PDF keeps where the glyphs sit and throws away what they meant. A scan is a harder case, because a scan does not even have glyphs. It has pixels. Everything else is reconstructed by software that is guessing, and the guessing is much better than it used to be, which is exactly what makes it dangerous.

Twenty-three years of setting scientific books taught me what a table actually is. It is not a picture of rows and columns. It is a grid of relationships: this number belongs to this row label and this column header, and it means nothing at all detached from either. Publishers rebuild tables by hand when they come in as images, and they do it because there is no honest shortcut.

That is the whole argument of this post. OCR converts pixels into characters. It does not convert a page back into a document.

OCR errors are not random, and that is the problem

If OCR failed noisily, this would be a solved problem. You would see garbage, you would know.

It does not fail that way. It fails toward plausibility. A 6 becomes an 8. A 1 becomes a 7. A 5 becomes an S. A comma becomes a full stop, which in a European financial document turns 1.234 into 1,234 or the reverse. The engine is choosing the most likely character given the shape, and the most likely character is almost always a real one.

In prose, this is survivable. You read a sentence with one wrong letter and your brain corrects it without stopping. That is why OCR feels so accurate: you are testing it on text where you are the error correction.

In a table of figures there is no such redundancy. Every digit is load-bearing. Nothing in the surrounding context tells you that a margin should be 6.2 rather than 8.2. The wrong number is exactly as readable as the right one, carries the same confidence, and looks equally like a fact.

Where scanned tables actually break

Four failure modes, and I have seen all four in real documents.

Column drift. A value lands one column to the left or right of where it belongs. Revenue for 2024 is now sitting under 2023. The total still adds up, because the same numbers are present, just attributed to the wrong period. Nothing looks wrong until someone builds a growth rate from it.

Tables that continue across pages. In a printed book this is handled by repeating the header and adding a continuation rule. OCR sees two separate blobs on two separate pages, and unless something explicitly stitches them, half the table silently disappears or becomes its own orphan table with no headers at all.

Merged and spanning cells. A header that spans three columns, or a row label that covers two rows, is a relationship expressed purely through geometry. Flatten it and the relationship is gone. The value that belonged to all three sub-columns now belongs to one of them, or to none.

Rule lines read as characters. A thin vertical rule becomes a 1 or a pipe. A horizontal rule becomes a row of dashes or underscores that the parser treats as content. This one is easy to spot when you look and impossible to spot when you do not.

The confidence score will not save you

Most OCR engines return a confidence value, and it is tempting to gate on it. Below some threshold, flag; above it, trust.

It does not work, for the same reason it does not work in retrieval. The confidence is per-character and it measures how sure the engine is about the shape it saw, not whether the resulting document makes sense. A crisp, clean, high-confidence 8 that was actually a 6 scores beautifully. A slightly smudged but correctly read digit scores worse. The score is orthogonal to the thing you care about.

I measured a version of this problem in a different context and published the result: retrieval similarity scores could not separate answerable questions from unanswerable ones, because the two populations overlapped. The lesson is the same here. Anything computed before you have the answer cannot tell you whether the answer is right. The check has to happen afterwards, against the source.

What to do instead

The habit is the same one that governs everything else I build. Treat the scan as evidence of a document, not as the document.

Keep the page image, always. If a figure is ever questioned, the answer is not a log line, it is the picture of the page it came from. This is the scanned-document version of a citation.

Rebuild tables as tables, and keep them that way through the entire pipeline. A table flattened into prose has lost the row-to-column relationship, which was the only reason the table existed.

Require every number to be findable in the source. Not paraphrased, not approximately present. Findable. If it cannot be located verbatim on the page it claims to come from, it does not get repeated.

Say the input was a scan. Put it in the output where the reader sees it, not in a log nobody opens. A brief built from clean born-digital PDFs and a brief built from a photocopied fax deserve different levels of trust, and the reader is entitled to know which one they are holding.

Never repair a number quietly. If something looks wrong, flag it and show the source. Correcting it invisibly means the next person inherits your guess as a fact.

Why this matters outside publishing

Publishers already know all of this. They budget for it, they employ people who rebuild tables by hand, and they treat a scanned manuscript as a cost rather than a convenience.

The people who get hurt are the ones who assume a PDF is a PDF. An acquirer reading management accounts that were scanned from a printout. A lender working from a photographed set of statements. Anyone whose most important number arrived as an image of a number.

The document you were sent and the document your software read are not always the same document. With a born-digital file, the gap is small. With a scan, it can be a decision made on a digit that was never there.

Cite the source or cut the claim. And when the source was a photograph of a page, say so out loud.