Pdf Processor
Unclaimed table debris poisons every text heuristic downstream
TLDR
When table detectors miss borderless grids, unclaimed numeric rows default to thePARAGRAPH type. Downstream heuristics (e.g. continuation linkers) mistake these numeric strings ("19 11 10 % 13 20 12") for prose sentences. Implementing localized line-level letter density gates (letters / nonSpace >= 0.35) and fragment-level digit ratio vetoes (digits > letters * 0.5) blocks invalid table debris merges without altering core classifier pipelines.
| Filtering Gate | Signal Condition | Action Taken |
|---|---|---|
| Line-Level Prose Gate | $\text{letterRatio} < 0.35 \text{ OR } \text{letters} < 3$ | Veto continuation link |
| Fragment-Level Digit Veto | $\text{digitCount} > \text{letterCount} \times 0.5$ | Veto continuation link |
Technical gate implementation
// Localized Prose Verification & Digit-Heavy Veto Gates
export function isProseLine(text) {
const letters = (text.match(/[A-Za-zÀ-ÿ]/g) || []).length;
if (letters < 3) return false;
const nonSpaceLength = text.replace(/\s+/g, '').length || 1;
return (letters / nonSpaceLength) >= 0.35;
}
export function isDigitHeavyFragment(text) { const digits = (text.match(/[0-9]/g) || []).length; const letters = (text.match(/[A-Za-zÀ-ÿ]/g) || []).length; return digits > letters * 0.5; }
export function validateSeamForLinking(prevLastLineText, nextFragmentText) { if (!isProseLine(prevLastLineText)) return false; if (isDigitHeavyFragment(nextFragmentText)) return false; return true; }
Rule of thumb: Verify prose density and digit ratio limits at the point of consumption rather than assuming default PARAGRAPH tags represent natural language.
Read this post in the full Engineering Journal →