When word-wrapped prose looks exactly like a table
TLDR: PDF files sometimes encode prose paragraphs as individual word-level text spans. A borderless table detector that clusters item X positions will find dozens of "column anchors" in those word positions. The standard fill-rate gate was supposed to reject this, but a split-and-retry path let a 3x100% fill-rate candidate through. The fix was an anchor quality gate that checks how many distinct columns show up consistently across rows.
The assumption that seemed reasonable
The stream table detector uses fill rate as its primary quality gate: tagged.length / (participating.length * colAnchors.length). In plain terms: what fraction of the cells in the detected grid are actually occupied?
A real table fills most of its cells. A false positive from random text should fail this gate because the "anchors" are sparse coincidences, not consistent columns.
This assumption held on the main test corpus. It held on financial reports, academic papers, and technical manuals. It failed on an HVAC installation manual where the right column contained paragraphs stored as word-level spans.
When it failed
The PDF encoding was: each word in the paragraph was a separate text item with its own X position. "with a CO Alarm(s). The Consumer Product Safety Commission" was 9 separate items, one per word.
The detector grouped these into Y-bands (one per text line), then clustered item X positions into anchors. With words landing at different X positions per line, the result was about 10 anchors across 15 bands. Fill rate: ~38 items / (15 * 10) = 0.25. That's below the 0.30 threshold. The candidate should have been rejected.
But it wasn't. The debug trace showed the rejection happening at the wrong level.
What was actually wrong
The full-page group had fill rate 0.20 and was correctly rejected. But the code had a fallback path where, if column anchors span a clear gap, it splits the bands at that gap and runs the candidate logic on each side independently. This path ran on the mixed 22-band group and the right-zone split came out with:
- 9 anchors, 15 participating bands
- Fill rate: 0.304 (just over 0.30)
zoneMode: true(from the split path, which always passes zoneMode)
Fill rate 0.30 on 15 bands with 9 anchors means roughly 41 items across 135 cells. That's legitimately dense. But the density came from a single pattern: one left-margin anchor (the paragraph indent position) appearing in 14 of 15 bands, and 8 noise anchors each appearing in exactly 2 bands.
What was actually wrong (structural version)
The fill rate gate is a ratio. It measures total item density across the grid. It does not care whether that density is evenly distributed across columns or concentrated in one.
A paragraph stored as word spans produces exactly this pathology: one dominant anchor (the line start position, where every line begins) plus many minor anchors (positions where words happen to coincide across lines). Fill rate looks healthy because the dominant anchor inflates the item count. Column consistency looks terrible because most anchors only appear in 2 rows.
The gate was measuring the wrong thing.
What got deleted
The _splitAnchorsAtXGap sentinel path (which returns { _xZoneSplit: splitX } from _buildCandidate) was kept because it handles legitimate use cases. But the relaxed gate thresholds applied to the resulting sub-candidates were too permissive. The key problem wasn't the split path itself, it was the absence of a column quality check.
What replaced it
An anchor quality gate after the anchor count check:
// Require at least half the anchors to appear in a meaningful fraction of bands.
// For small groups (<=5 bands), anchors must appear in ALL bands.
// For larger groups, require >= 25% band presence.
const minPresenceBands = bands.length <= 5
? bands.length
: Math.max(3, Math.floor(bands.length * 0.25));
const anchorBandCounts = colAnchors.map( a => new Set(a.items.map(i => i._band)).size ); const qualifiedAnchors = anchorBandCounts .filter(bc => bc >= minPresenceBands).length;
if (qualifiedAnchors < Math.ceil(colAnchors.length / 2)) return null;
For the false-positive case: 9 anchors, 16 bands. minPresenceBands = max(3, floor(16 * 0.25)) = 4. Anchor band counts: [14, 2, 2, 2, 2, 2, 2, 2, 2]. Anchors with bc >= 4: 1. Required: ceil(9/2) = 5. Rejected.
For a real table: anchor counts are roughly evenly distributed. A 7-row table with 3 columns would have each anchor appearing in 5-7 rows. All 3 qualified, need 2. Passes easily.
The small-group edge case
A separate false positive appeared on a different page: 3 bands (exactly the minimum), 5 anchors, fill rate 0.87. All 5 anchors appeared in at least 2 of the 3 bands. The anchor quality gate didn't help because with only 3 bands and word-span prose, any 3 dense lines align well.
The fix for small groups: when bands.length <= 5, require anchors to appear in ALL bands (not just 25%). This forces the minimum 100% column presence check at the lower bound where the noise-to-signal ratio is worst.
The content in that 3-band case was: "NOT | operate." / "and | phase | correspond | to | that" / "handle | load | imposed | by | this". Three sentence fragments from a paragraph, not a table. The row spacing was irregular (gaps of 135 px and 45 px). But confidence was 0.64 because column alignment was tight (word-span items start at exact positions).
The generalizable lesson
Gate metrics based on averages or ratios can look healthy even when the underlying distribution is pathological. Fill rate is an average. An average over 135 grid cells tells you nothing about whether any one column appears consistently.
Before adding another ratio gate, ask: can this ratio be satisfied by a concentrated distribution that a real table wouldn't have? If yes, add a distribution check alongside it. Anchor band counts is a distribution check: it forces consistency across the column dimension, not just density across the whole grid.
The two together (fill rate + anchor quality) reject the patterns that fool each individually.