Histograms measure the wrong thing when you feed them sub-unit deltas

TLDR

Histogramming vertical $Y$-deltas across raw PDF text items produces skewed modal values because PDF.js emits separate text items per style run and word fragment. Intra-line Y-jitter ($0.1\text{ to }1.0\text{px}$) far outnumbers true inter-line gaps ($24\text{px}$), dragging histogram modes toward sub-pixel noise. Deduplicating text items into distinct baseline Y-coordinates before calculating inter-line deltas ensures that histogram modes accurately reflect true document line pitch.

Histogram Data Input	Dominant Histogram Peak	Calibrated Line Pitch Output	Paragraph Paragraph Splitting Result
Raw Consecutive Text Items	Intra-line Y-jitter ($0.5\text{px}$)	$5.4\text{px}$ (Skewed mode)	Fails (Every line becomes a paragraph)
Deduplicated Baseline Y-Lines	Inter-line pitch ($24.0\text{px}$)	$26.0\text{px}$ (Accurate mode)	Passes (Paragraphs join correctly)

---

Technical defect analysis

Text runs on the same visual line frequently have slightly different baseline coordinates due to PDF rendering precision:

Item 1 ("We introduce"): vy = 120.1px
Item 2 ("a new language"): vy = 120.4px  --> dy = 0.3px (Intra-line jitter)
Item 3 ("representation"): vy = 144.2px  --> dy = 23.8px (True inter-line gap)

// DEFECTIVE IMPLEMENTATION: Delta histogram across raw item runs
const sortedItems = [...textItems].sort((a, b) => a.vy - b.vy);
const deltas = [];
for (let i = 1; i < sortedItems.length; i++) {   const dy = sortedItems[i].vy - sortedItems[i - 1].vy;   if (dy > 0 && dy < 80) deltas.push(dy); // Intra-line jitter (0.3px) overwhelms 24px peaks! }

Because intra-line jitter samples occurred far more frequently than inter-line gaps, histogram mode calculations reported a line pitch of $5.4\text{px}$ instead of $24.0\text{px}$.

---

Remediation: baseline Y-deduplication before delta histogramming

Collapse text items into unique integer baseline Y-coordinates before computing line-to-line deltas:

export function calculateDocumentLinePitch(textItems) {
  // Step 1: Round and deduplicate Y-coordinates to extract distinct baseline rows
  const uniqueYLines = Array.from(new Set(textItems.map(item => Math.round(item.vy))))
    .sort((a, b) => a - b);
// Step 2: Compute deltas strictly between distinct baseline rows   const deltas = [];   for (let i = 1; i < uniqueYLines.length; i++) {     const dy = uniqueYLines[i] - uniqueYLines[i - 1];     // Enforce a minimum floor (dy >= 4px) to exclude residual rounding noise     if (dy >= 4 && dy < 80) {       deltas.push(dy);     }   }
if (deltas.length === 0) return null;
// Step 3: Compute mode over clean baseline-to-baseline deltas   const modalPitchPx = computeHistogramMode(deltas);   return refinePitchByMedian(deltas, modalPitchPx); }

Verification output

• Before Fix: leadingPx = 5.4px (Paragraph gap cutoff derived as $8.1\text{px} \rightarrow$ every line split into separate <p>).
• After Fix: leadingPx = 26.0px (Paragraph gap cutoff derived as $39.0\text{px} \rightarrow$ sentences joined into clean paragraphs).

Rule of thumb: Aggregate raw spatial data into conceptual units (distinct baseline Y-lines) before computing delta distribution histograms.