Engineering Journal
Pdf Processor
Pdf Processor

We shipped a parser where 96% of the output could not be pointed at

2026-08-21

TLDR: Our extractor stamped an addressable id on tables and pictures only. Everything else, paragraphs, equations, callouts, headings, page furniture, rendered fine and could not be located afterward. On one test document, zero of 315 regions were addressable. The assumption that broke us was that ids are an implementation detail of the types that need them.

The assumption that seemed reasonable

When you extract a table, you need to find it again: someone clicks "send this table to the spreadsheet tool" and you have to resolve that click back into rows. Same for a picture. So the code stamped an id where the id was needed:

html = <div class="table-wrap" data-region-id="${region.id}">${tableHtml}</div>;

A paragraph seemed different. Nobody sends a paragraph anywhere. It renders, you read it, done. Adding an id to something with no consumer looked like speculative work, and we have all been told not to do speculative work.

That reasoning is fine right up until the moment the system grows a feature that treats every region uniformly. Then "the types that needed ids" turns out to have been a snapshot of one week's requirements.

When it failed

We added an artifacts panel: every extracted region across every page becomes a selectable item, click to jump to it, select several and send them onward. It worked in the demo.

It worked in the demo because the demo document was a report full of tables.

The lookup was page-scoped and correct:

getRegionHtml(page, regionId) {
    const scope = doc.querySelector(section[data-page="${page}"]);
    return scope?.querySelector([data-region-id="${regionId}"])?.outerHTML ?? null;
}

Correct, and returning null for four region types out of seven. Clicking a paragraph did nothing. Selecting a callout and sending it produced an empty payload. The failure was silent in the worst way: the panel listed the artifact, so the artifact appeared to exist.

Then we ran two engineering papers through it. One produced twelve addressable regions out of roughly three hundred. The other produced zero. It had no tables the detector recognized and no raster images, so nothing in a fourteen-page document could be pointed at. Every tag in the panel was a link to nowhere.

What was actually wrong

Not the lookup. Not the panel. The mistake was deciding addressability per type, at the leaf, in seven different places.

Every region already passed through one common wrapper on its way to markup:

return <div class="region" data-ry="${ry}" data-rx="${rx}">${html}</div>;

Every single region, whatever its type, got that wrapper. It carried geometry. It did not carry identity. Identity was being decided seven lines further down, inside per-type branches, by whoever wrote each branch.

That is the real defect: the property was universal and the decision was local. Nobody wrote "paragraphs are not addressable." Seven people independently wrote code for one type each, and two of them happened to need an id.

What got deleted

Nothing, structurally. That is worth saying, because a bug this bad suggests a rewrite, and the fix was one attribute in one place:

const idAttr = region.id != null
    ?  data-region-id="${esc(String(region.id))}" data-region-type="${esc(region.type)}"
    : '';
return <div class="region"${idAttr} data-ry="${ry}" data-rx="${rx}">${html}</div>;

Addressability moved from seven local decisions to one structural fact. On the same two documents: 12 addressable regions became 315, and 0 became 315.

The type came along for free, and it turned out to matter as much as the id. The consumer that reads the markup back into structured form had been guessing at what each element was from its CSS classes. Now the wrapper says so.

What replaced it

One rule, and a consequence.

The rule: if a property is true of every member of a category, implement it where the category is handled, not where the members are. The wrapper is where "this is a region" is expressed, so it is where "regions have identity" belongs.

The consequence: the consumer had to learn to look up. The importer read ids off the leaf element, which used to be where they lived. Now the id is on the wrapper and the walk descends through it, so the leaf has to search upward:

const existing = el.getAttribute('data-region-id')
    || el.closest('.region[data-region-id]')?.getAttribute('data-region-id')
    || null;

That introduced a second bug immediately, and it is the interesting one. A single wrapper can contain several leaf elements, because the text rebuilder emits a paragraph per sentence-aware break. All of them now found the same id by looking up. Two blocks sharing one id is exactly as broken as no id at all: the lookup resolves both to whichever comes first. The fix was a per-page set of claimed ids and a suffix for repeats.

Uniqueness is not a property you get from stamping. It is a property you get from checking.

The generalizable lesson

Ask which properties of your output are universal and which are per-type. Universal properties implemented per-type will be complete only for the types someone was actively working on, and the gap will be invisible because the working types are the ones in every demo and every test fixture.

Two questions catch this class before it ships:

  1. Would a document containing none of the types I built for still work? Our answer was no, and nobody asked, because every test document had tables in it.
  2. Is this property decided in one place or in N places? N places means N chances to forget, and the ones you forget will be the ones you do not use yourself.
The panel was not broken. The panel was asking a reasonable question of output that could not answer it.
Read this post in the full Engineering Journal →