A Containment Check Passes Because the List It Checks Is Empty

TLDR

Bug: spurious divider elements appearing inside paragraph regions. Cause: divider detection ran before text regions were created, so the "already inside a region" containment check always returned false. Fix: move the classifier to after the stage that produces the data it depends on.

Repo: tools/pdf-processor

Symptom

Extracted HTML contained unexpected horizontal rule elements inside what should have been paragraph text. On documents with decorative horizontal lines near body copy, lines that visually separated sections were being inserted mid-paragraph instead of between blocks.

The symptom was document-dependent. Documents without horizontal rules inside body text showed no problem. The bug was not caught in testing because the test documents did not exercise the case.

Why It Happens

Containment checks in pipelines are only as correct as the data they query. If a stage checks "is this item already claimed by a region?" but runs before the stages that produce the regions it should be checking, the check will always return false.

The relevant check:

function detectDividers(segments, regions) {
  return segments.hLines.filter(seg => {
    const inside = regions.some(r => containsBbox(r.bbox, seg));
    return !inside; // claim this segment as a divider if unclaimed
  });
}

The logic is correct. A horizontal segment that falls inside a known region should not be classified as a divider. But this function ran before the text classification stage that creates paragraph, heading, and list regions. At the time it ran, regions only contained image, table, and box entries.

Every horizontal segment inside a paragraph area tested as unclaimed. Every one became a divider.

// wrong ordering: dividers run before text classification
const dividers = detectDividers(segments, regions);  // regions: no text yet
regions.push(...dividers);
const text = classifyText(items, columns);           // regions: paragraphs added here regions.push(...text);

The Fix

Move the classifier to after the stage that produces the data it depends on.

// correct ordering
const text = classifyText(items, columns);
regions.push(...text);                               // paragraphs now exist
const dividers = detectDividers(segments, regions);  // containment check works correctly regions.push(...dividers);

The algorithm did not change. The position in the sequence changed. One line moved.

How to Prevent It

Write the pipeline sequence as explicit function calls in an orchestrator, not as sections of a monolithic function. When ordering is visible as a call sequence, a misplaced call is a visible code smell. When ordering is implicit in the position of code inside a large function, there is no structural signal that something is in the wrong place.

For any classifier that queries a shared data structure, add a comment at the call site naming the dependency:

// must run after classifyText(): depends on paragraph/heading regions existing
const dividers = detectDividers(segments, regions);

This is the diff that makes a future reorder visible as a constraint violation rather than a silent behavior change.

The Generalizable Lesson

A correctness check on shared state is only correct if the state is fully populated when the check runs. Verify that all stages that contribute to a shared structure have completed before any stage that reads from it.