Column contamination in borderless table detection

TLDR: Borderless table detection reads column positions from text alignment. If you run it on a two-column page without splitting the columns first, the left and right columns contaminate each other's anchors and every gate metric becomes meaningless. The fix is a pre-pass column split before the detector runs, not a change to the detector itself.

The problem class

Borderless (stream) table detection works by clustering item X positions into column anchors, then measuring how consistently those anchors repeat across rows. The core bet is that tabular data has a different X distribution than prose: tables have a few X positions that appear in most rows, prose has many X positions distributed across the line.

That bet is only valid when all the items being analyzed belong to the same textual region. When a page has two columns and you run the detector on both together, you get:

• Anchors from the left column mixed with anchors from the right column
• Fill rate computed against a combined anchor set that no real row satisfies
• Row spacing computed across rows from different content streams

Every metric degrades. The detector either misses tables that should be found or fires on spans of text that share nothing but an accident of page geometry.

Why this is easy to miss

The obvious place to call the detector is after the lattice pass, on whatever text is still unclaimed. That's exactly the right tier: stream detection should only see text that wasn't grabbed by a bordered table. The mistake is that "unclaimed text" is not the same as "text that belongs to one content stream."

A two-column academic paper has unclaimed prose in both columns after the lattice pass. The column boundaries don't filter out automatically. The detector sees interleaved items from two independent columns and tries to make sense of them as a single table candidate.

On a page with no real borderless table, this corrupts all the metrics in the direction of rejection (high fill rate, high item count, bad row spacing) and the detector correctly returns nothing. On a page that has a borderless table in one column, the contamination is lethal: the table's clean anchor signal gets diluted by prose from the other column, fill rate drops, and the table is missed.

The naive fix and why it fails

The first attempt was to detect a page-column gap inside the detector itself. The idea: if the column anchors span a gap that looks like a page column boundary (the largest gap is much bigger than the median gap), split the anchors at that gap and run the candidate logic on each side independently.

This was implemented as _splitAnchorsAtXGap: split only if the largest gap is >= 3x the median AND >= an absolute pixel floor. The thresholds were conservative to avoid splitting genuine intra-table whitespace (a "System" column separated from score columns by 100+ px is not a page column boundary).

The problem: on the test page where both columns had unclaimed text, the gap between left-column anchors and right-column anchors was 125 px. The absolute floor was 144 px. The ratio was 2.3x, not 3x. It was just below both thresholds, so no split.

The thresholds couldn't be tightened without false-positives on dense tables, and couldn't be relaxed without false-splits on sparse tables. The approach was fundamentally reactive: trying to detect column structure from anchor statistics that were already contaminated.

The fix: pre-split before detection

The right place to resolve column structure is before any anchor clustering happens, not after. The contextClassifier already has a column detection step for the main layout pass. The same function runs on unclaimed text before stream detection and produces reliable split X positions.

// In the orchestrator, before stream detection runs:
let streamColXs = [];
if (unclaimedMeta.length > 10) {
    const { splits: preSplits } = detectPageColumns(unclaimedMeta, viewport, scale);
    streamColXs = preSplits
        .map(s => s.x ?? s)
        .filter(x => x > viewport.width  0.1 && x < viewport.width  0.9);
}
const streamTables = detectStreamTableRegions(
    unclaimedMeta, scale, regions, tableSegs, pageGraph, streamColXs
);

When columnXs is non-empty, the detector runs separately per zone:

export function detectStreamTables(
    textMeta, scale, latticeRegions = [], segments = [], columnXs = []
) {
    if (columnXs.length > 0) {
        const boundaries = [-Infinity, ...columnXs, Infinity];
        const results = [];
        for (let zi = 0; zi < boundaries.length - 1; zi++) {
            const lo = boundaries[zi], hi = boundaries[zi + 1];
            const zoneItems = textMeta.filter(i => i.vx >= lo && i.vx < hi);
            if (zoneItems.length < 6) continue;
            const zoneResults = detectStreamTables(
                zoneItems, scale, [...latticeRegions, ...results], segments, []
            );
            for (const r of zoneResults) {
                if (!_overlapsLattice(r.bbox, results)) results.push(r);
            }
        }
        return results;
    }
    // ... single-zone detection path
}

Each zone call passes columnXs = [], so recursion terminates. Results from earlier zones are added to the overlap exclusion list for later zones, preventing duplicates at zone boundaries.

Zone mode effects on gate thresholds

When running inside a zone, two relaxations are appropriate that would be too aggressive on full-page mixed text:

Secondary adaptive gap split. The standard band grouping uses 2.5x the median inter-band gap as a split threshold. In zone mode, a tighter secondary split fires when a compact cluster of 4+ bands (tight pitch, short average text) is followed by a gap >= 2x the cluster's own median pitch. This catches tables that sit immediately above prose sections where the table-to-prose gap is smaller than the global 2.5x threshold.

avgLen relaxation. Tables with long row labels (system names, model names, condition descriptions) would fail a strict average-length cap. In zone mode, when the detector has already found gutter evidence (clear vertical blank space across bands), the avgLen cap is raised from 20 to 40 characters. This is safe in zone mode because the items are already filtered to one column, so a high avgLen can't mean "this is a long prose sentence."

const avgLenCap = (zoneMode && hasGutterEvidence)
    ? Math.max(scale.STREAM_MAX_AVG_LEN, 40)
    : scale.STREAM_MAX_AVG_LEN;
if (avgLen > avgLenCap) return null;

Test corpus results

The column split pre-pass was the fix for a research paper (N19-1423) where pages 7 and 8 had borderless tables in a two-column layout. Before the fix: 0 stream tables detected on both pages. After: 1 stream table per page, covering the correct bbox.

The same fix also eliminated 5 false-positive stream tables that had appeared on the same paper because the detector was finding spurious "tables" that spanned both columns.

When to apply this pattern

Any region detector that relies on spatial statistics (alignment, density, distribution) should be pre-filtered to a spatial domain where those statistics are meaningful. For text layout, that domain is the page column. Running on the full page is only correct when the page has one column.

The cost is that column detection must run before table detection, and it must run on unclaimed text (not all text, since claimed text may distort the column signal). The benefit is that every gate metric becomes trustworthy because the items analyzed belong to a coherent content stream.