Unclaimed table debris poisons every text heuristic downstream

TLDR: A paragraph-continuation linker started chaining table rows into prose: "19 11 10 % 13 20 12" linked to "Net sales mix:". The bug class is classifier fallthrough: content one detector missed gets a default type, and downstream heuristics trust the type. The fix is cheap prose and digit gates at the point of consumption.

The bug class

Pipelines that classify content and then process it by type all share this failure mode. Detection is imperfect, so some content falls through to the default type.

In document extraction the default type is "paragraph." A table the table detector missed does not disappear. Its rows become paragraph regions, and everything downstream that trusts the PARAGRAPH label now operates on numbers pretending to be sentences.

The consumer that exposed it here was a continuation linker joining paragraph fragments across column seams. Its signals (no terminal punctuation, flush left edge, same font) are all true for a row of table numbers. So it happily produced:

"19 11 10 % 13 20 12"  ⟶  "Net sales mix:"
"1"                    ⟶  "0.5"        (chart axis ticks)

Why the pipeline produces it

The type system lies at the edges. PARAGRAPH does not mean "this is prose." It means "no structural detector claimed this."

Heuristics tuned on real prose implicitly assume the label is truthful. The failure is invisible in unit tests because test fixtures are built from content that classifies cleanly.

The fix

Do not trust the label at the consumption point. Verify the property you actually need, which is "reads like running text," with two gates.

Line-level, on the exact lines at the seam:

function isProse(text) {
  const letters = (text.match(/[A-Za-zÀ-ÿ]/g) || []).length;
  if (letters < 3) return false;
  const nonSpace = text.replace(/\s+/g, '').length || 1;
  return letters / nonSpace >= 0.35;
}

And fragment-level, because a seam line can read as prose while the fragment is a table ("Consolidated 10 14 11 12" passes the line gate on the strength of one word):

function digitHeavy(text) {
  const digits  = (text.match(/[0-9]/g) || []).length;
  const letters = (text.match(/[A-Za-zÀ-ÿ]/g) || []).length;
  return digits > letters * 0.5;
}

The 0.5 ratio has headroom on both sides. Real prose, even financial prose full of dollar figures, stays well under it. Table rows blow past it.

Both gates are vetoes, not scores. Content that fails them is never linked, whatever the other signals say.

Preventing the class

Two habits generalize. First, at every consumer of a classified type, ask what property you are actually depending on, and check that property when the check is cheap. A regex ratio costs nothing next to the corruption it prevents.

Second, treat these gate hits as telemetry. Every fragment the prose gate rejects is a detector miss upstream. In this codebase the rejected fragments map one-to-one onto known table-detection gaps, which turned the workaround into a coverage worklist for the real fix.

The lesson

A default type is not a classification, it is an absence of one. Heuristics that consume it must verify the property they need instead of inheriting the label's optimism.