Why Reliable Document Extraction Starts With an Evidence Graph

The primary reader is any developer building a document parser that must do more than return a blob of text. The useful lesson is how to introduce layout models and language models without letting either erase what the source file actually contained.

The problem class

A PDF extractor observes several imperfect versions of one document. The content stream contains glyphs and paths but may not describe paragraphs or tables. OCR sees pixels but can miss characters and reading order. A layout detector finds semantic regions but does not know which source characters belong to them. A language model understands headings and forms but can confidently invent structure.

The common mistake is to make one of those systems authoritative. That creates a pipeline where later stages overwrite earlier output. By the time a user notices a wrong number in a table, the system cannot explain whether it came from the PDF, OCR, box assignment, or an AI rewrite.

The naive pipeline

A typical implementation looks linear:

const text = await extractText(pdf)
const regions = await detectLayout(render(pdf))
const markdown = await llm.rewrite({ text, regions })
return markdown

This is easy to demonstrate and difficult to trust. The final Markdown is detached from the source geometry. There is no durable representation of disagreement. Running a new model changes the entire output, and validation can only compare strings after the fact.

A better model: observations before interpretations

An evidence graph stores immutable observations first. A text span, painted path, image, widget, OCR token, or detector box becomes a record with a stable ID, coordinate space, source, tool version, and confidence when confidence is genuinely measured.

const evidence = {
  id: 'ev:p3:text:41',
  page: 3,
  kind: 'text-span',
  value: 'Net income',
  bbox: { space: 'pdf-points', x: 72, y: 418, w: 66, h: 9 },
  source: 'pdf-content-stream',
  confidence: null,
}

The null matters. Native text is not a probabilistic guess, while some sources simply do not expose a calibrated score. Replacing unknown confidence with 0.7 makes dashboards look complete while corrupting the trust model.

Interpretations live in a second graph. A table region can reference line evidence, contained text spans, and a detector box. A form field can reference an AcroForm widget, its label span, and a containment rule. None of those operations mutate the observations.

const region = {
  id: 'p3:table:2',
  type: 'table',
  evidenceIds: ['ev:p3:path:9', 'ev:p3:text:41', 'ev:p3:model:5'],
  status: 'inferred',
  relations: [{ type: 'contains', target: 'p3:cell:1' }],
}

Why relationships matter more than more boxes

Detector benchmarks reward boxes that overlap labeled boxes. Document consumers need attachments: captions belong to figures, cells belong to tables, fields belong to forms, and text belongs to reading-order sequences. A detector can improve mean average precision while leaving more source text orphaned.

Relationships make structure explicit. They also let deterministic geometry do most of the work. Containment fraction, alignment residuals, scale ratios, reading-order gaps, and widget membership are strong features that cost no learned parameters. A small learned relation scorer can resolve the ambiguous cases without duplicating a large detector.

The result resembles a compiler more than a chatbot. Evidence is the token stream, resolved regions are an intermediate representation, constraints are semantic analysis, and HTML or Markdown are target formats.

Adding AI without surrendering authority

The language model should see only a failed or uncertain subgraph. It receives stable IDs, available operations, and constraints. Instead of returning a rewritten document, it returns a patch:

{
  "op": "associate_label_value",
  "labelId": "p3:text:12",
  "fieldId": "p3:field:4",
  "reason": "nearest aligned label ending in a colon"
}

Deterministic code then checks that both IDs exist, the relation is legal, geometry is plausible, no containment cycle is introduced, and a named quality measure improves. The proposal may be accepted, rejected, or escalated. Either way, it is recorded separately from the source evidence.

This division keeps the model useful. It can recognize semantic patterns that would be expensive to encode as rules, but it cannot silently replace a number, invent a region, or erase conflicting evidence.

Runtime independence

An evidence graph also creates a clean browser/server boundary. Browser PDF APIs, server workers, and different inference runtimes may collect evidence differently, but they emit the same contracts. The resolver and emitters then operate over a stable representation.

Parity should be semantic rather than accidentally byte-for-byte. Floating-point values may vary slightly across runtimes; element identity, text, relationships, coordinate meaning, and provenance may not.

Verification becomes much more precise

Once the intermediate representation is explicit, tests can target the layer that failed. Evidence fixtures verify glyphs, operators, transforms, and widgets without depending on HTML. Resolver fixtures assert parents, ordering, and table spans without rerunning OCR. Emitter fixtures verify that a correct graph survives conversion. End-to-end tests still matter, but they no longer have to diagnose every failure from one enormous snapshot.

Visual debugging improves too. An overlay can draw evidence boxes in one color, resolved regions in another, and relationship edges between them. Selecting an exported paragraph can highlight the exact source spans and operations that created it. Coordinate mistakes, class-order errors, and nonsensical cross-column attachments become immediately visible.

The graph also supports targeted re-extraction. If OCR improves on page twelve, the system can replace that page's OCR evidence, invalidate dependent interpretations, and replay the resolver. It does not need to regenerate trusted native pages or discard user corrections. Content hashes make caching safe because a cached result is tied to both its source bytes and the configuration that produced it.

Tradeoffs

The graph costs memory and requires schema discipline. Stable IDs must survive revisions. Coordinates need declared spaces. Operations require versioning. Debugging tools must render not only boxes but also relationship edges and supporting evidence.

Those costs replace a worse cost: untraceable output that cannot be corrected safely. Once every output element can explain its origin, model upgrades become controlled graph revisions instead of full-document rewrites.

The transferable principle is simple: preserve observations, separate interpretations, and make every probabilistic improvement arrive as a reviewable operation. That is the foundation of extraction software people can use for consequential documents.