The fill rate gate that let prose through

TLDR: Stream table detection uses fill rate to reject false positives, but fill rate measures average cell density across the entire grid. A dominant left-margin anchor plus several coincidental word-position anchors can produce a fill rate above the threshold while no real column structure exists. Fix: require that at least half the column anchors appear consistently across rows, not just in 2 bands.

The bug class

PDF encoders sometimes produce word-level text spans instead of whole-sentence items. Each word becomes a separate text item with its own X position. A stream table detector that clusters X positions will find many "anchors" in those word positions.

The fill rate gate is supposed to catch this: tagged.length / (participating.length * colAnchors.length). If most "cells" in the detected grid are empty, fill rate drops and the candidate is rejected.

But when a paragraph has one consistent left-margin position (every line starts there) and several coincidental word positions (words that happen to align across a few lines), the item count can be high enough for fill rate to pass.

The diagnosis

Debug trace on a false-positive stream table:

bands=16 anchors=9 participating=15 fillRate=0.304
anchor band counts: [14, 2, 2, 2, 2, 2, 2, 2, 2]

Anchor at position x=632 appears in 14 of 16 bands. That's the paragraph left margin, where every line starts. The other 8 anchors each appear in exactly 2 bands. Those are random word-position coincidences.

Fill rate: ~41 items / (15 * 9 cells) = 0.304. Just over the 0.30 threshold. Confidence: 0.66.

The left-margin anchor inflated the item count enough to push fill rate above the gate. But having one consistent column plus eight noise columns is not a table.

The fix

Add a column quality check after anchor discovery, before any gate metrics:

const minPresenceBands = bands.length <= 5
    ? bands.length
    : Math.max(3, Math.floor(bands.length * 0.25));
const anchorBandCounts = colAnchors.map(     a => new Set(a.items.map(i => i._band)).size ); const qualifiedAnchors = anchorBandCounts     .filter(bc => bc >= minPresenceBands).length;
if (qualifiedAnchors < Math.ceil(colAnchors.length / 2)) return null;

For the failing case: 9 anchors, 16 bands, minPresenceBands = max(3, floor(16 * 0.25)) = 4. Only 1 anchor has bc >= 4. Required: 5. Rejected.

For a real table with 7 rows and 3 columns: each anchor appears in 5-7 rows. All 3 qualify, need 2. Passes.

The small-group edge case

A second false positive had 3 bands (the minimum), 5 anchors, fill rate 0.87. All 5 anchors appeared in 2+ bands. This passed even with minPresenceBands = max(3, floor(3 * 0.25)) = 3 because max(3, 0) = 3 and the anchors each appeared in 2-3 bands.

For small groups, the threshold must be stricter: when bands.length <= 5, require anchors to appear in ALL bands. Three rows of prose can align at 5 word positions, but a real 3-row table would have every column present in all 3 rows.

How to prevent this class

Fill rate measures density. You also need a distribution check over the column dimension. Before shipping any candidate gate:

1. Pick a pathological input that should fail (prose, lists, ragged text)
2. Compute every gate metric on that input manually
3. Check whether the metric can pass on the pathological input through a concentrated distribution (one hot cell, one dominant anchor, one aligned row)

If yes, add a distribution check alongside the ratio gate. For column-anchor presence: check band counts per anchor, not just count of anchors.