Histograms measure the wrong thing when you feed them sub-unit deltas

TLDR

We calibrated line spacing by histogramming y-deltas between consecutive text items. Items on the same visual line differ by fractional jitter, and those near-zero deltas outnumbered real line gaps, dragging the mode toward noise. The fix: deduplicate items into lines first, then measure deltas between lines.

Symptom

A document-level "leading" statistic came out far below the visible line pitch. Downstream tolerances derived from it fragmented paragraphs into single-line blocks in the rendered output.

Nothing crashed and nothing logged. The number was just quietly wrong, which is the worst kind of wrong for a calibration value.

Why it happens

A visual line of text is not one item in pdf.js. getTextContent() returns a separate item per style run and often per word cluster, and their baselines differ by fractions of a pixel.

Sort all items by y and take consecutive deltas, and you get two populations: thousands of jitter deltas within lines, and hundreds of real line-pitch deltas between lines. The histogram mode goes to the bigger population. A dy > 0 filter does not save you, because jitter is positive too.

This shape shows up anywhere you histogram deltas of clustered values: scroll positions, timestamps from batched events, GPS points. If your samples come in clumps, consecutive-sample deltas measure clump density, not the spacing you care about.

The fix

// wrong: deltas between consecutive ITEMS, jitter dominates
const sorted = [...items].sort((a, b) => a.y - b.y);
for (let i = 1; i < sorted.length; i++) {
    const dy = sorted[i].y - sorted[i - 1].y;
    if (dy > 0 && dy < 80) deltas.push(dy);
}
// correct: collapse items into lines, then measure line-to-line const lines = [...new Set(items.map(it => Math.round(it.y)))].sort((a, b) => a - b); for (let i = 1; i < lines.length; i++) {     const dy = lines[i] - lines[i - 1];     if (dy >= 4 && dy < 80) deltas.push(dy); }

The dedup step converts the two-population sample into one population: gaps between distinct baselines. The dy >= 4 floor drops rounding survivors. After that, the histogram mode is the line pitch, because line pitch is now the only thing in the histogram.

Evidence

Calibration output on the same 16-page document, before and after the dedup (plus a median refinement within 20% of the mode):

before: leadingPx = 5.4     // jitter-adjacent, rejected as implausible upstream
after:  leadingPx = 26      // actual baseline-to-baseline pitch is ~24-27

With the corrected value, the paragraph-gap threshold derived from it (1.5x leading = 39px) correctly joins consecutive lines (24px apart) and splits at real paragraph gaps (40px+). The rendered document went from one block per line to properly joined paragraphs, verified in the app's visual diff view against the source PDF.

How to prevent it

Before histogramming deltas, ask what one sample represents. If the answer is "a fragment of the thing I care about," aggregate to the thing first.

A cheap invariant check also catches it: the mode of your deltas should be the same order of magnitude as a value you can eyeball. Line pitch in a rendered PDF is easy to eyeball at around 24px; a calibrator reporting 5.4 should have failed loudly.

The generalizable lesson

Delta statistics inherit the granularity of your samples, not the granularity of your concept; aggregate samples up to the concept before you measure.