Engineering Journal
Pdf Processor
Pdf Processor

Unclaimed table debris poisons every text heuristic downstream

2026-07-04

TLDR

When table detectors miss borderless grids, unclaimed numeric rows default to the PARAGRAPH type. Downstream heuristics (e.g. continuation linkers) mistake these numeric strings ("19 11 10 % 13 20 12") for prose sentences. Implementing localized line-level letter density gates (letters / nonSpace >= 0.35) and fragment-level digit ratio vetoes (digits > letters * 0.5) blocks invalid table debris merges without altering core classifier pipelines.
Filtering GateSignal ConditionAction Taken
Line-Level Prose Gate$\text{letterRatio} < 0.35 \text{ OR } \text{letters} < 3$Veto continuation link
Fragment-Level Digit Veto$\text{digitCount} > \text{letterCount} \times 0.5$Veto continuation link

Technical gate implementation

// Localized Prose Verification & Digit-Heavy Veto Gates
export function isProseLine(text) {
  const letters = (text.match(/[A-Za-zÀ-ÿ]/g) || []).length;
  if (letters < 3) return false;
  const nonSpaceLength = text.replace(/\s+/g, '').length || 1;
  return (letters / nonSpaceLength) >= 0.35;
}

export function isDigitHeavyFragment(text) { const digits = (text.match(/[0-9]/g) || []).length; const letters = (text.match(/[A-Za-zÀ-ÿ]/g) || []).length; return digits > letters * 0.5; }

export function validateSeamForLinking(prevLastLineText, nextFragmentText) { if (!isProseLine(prevLastLineText)) return false; if (isDigitHeavyFragment(nextFragmentText)) return false; return true; }

Rule of thumb: Verify prose density and digit ratio limits at the point of consumption rather than assuming default PARAGRAPH tags represent natural language.
Read this post in the full Engineering Journal →