Recovering stacked math from flat glyph rows

TLDR

PDF and OCR pipelines flatten stacked notation into a single baseline stream, so fractions, summation limits and radicals arrive as ordinary text items. Rebuilding the structure is not a pairing problem between atoms, it is a row classification problem first. Cluster glyphs into rows, measure gaps against the farthest absorbed right edge, and compute size thresholds over the same population the rule applies to. This post shows the algorithm that reassembles display math from flat glyph geometry and renders it with a locally vendored LaTeX renderer.

Problem class: the flattening pipeline

Any document parser that works from glyph positions, not from a semantic source, sees the same thing: 1 and 2 sitting at two different y coordinates with a rule between them. The human eye reads a fraction. The data model reads two text items on two baselines.

The naive approach is to pair atoms directly. Find a numerator, find a denominator, emit a fraction. This fails in three specific ways, and each failure is a different lesson.

Naive approach: per-atom pairing

The first instinct is a local rule. Take each atom, look for another atom above or below it, and when one exists, treat them as a fraction pair.

// Naive: pair every atom with whatever sits above or below it
function collapseFractions(atoms) {
  return atoms.map(atom => {
    const above = atoms.find(a => isAbove(a, atom));
    const below = atoms.find(a => isBelow(a, atom));
    if (above && below) return { frac: [above, atom, below] };
    return atom;
  });
}

This looks correct on a lone 1/2. It breaks on the first nested structure, and it breaks silently. An atom below your numerator may belong to a different fraction entirely, one row further down. A pairing rule cannot see that, because pairing never asked which row anything was in.

Better model: classify rows first

Structure is assigned to rows, not to atoms. A row is a set of glyphs that share a baseline. Fraction pairing is then a relationship between rows, and the atoms in each row are decided by row membership, not by local proximity.

The concrete rules that make this work:

1. Cluster full-size glyphs by baseline into rows, with a tolerance small enough that a fraction's two halves never merge.
2. Split rows at real horizontal gaps. A superscript hanging over a gap does not close it, so measure the gap against the farthest right edge of any atom inside the span, not against the previous atom's right edge.
3. Pair two rows as numerator and denominator only when no other full-size row sits in the band between them. That one rule is what keeps a/b + c/d from merging into one four-layer stack.
4. Recover nesting by pairing the tightest row pair first and recursing, so a fraction inside a fraction lands in the right place.

// Row-based: build rows, split on real gaps, pair tightest first
function buildRows(atoms, baseSize) {
  const rows = clusterByBaseline(atoms, 0.3 * baseSize);
  for (const row of rows) splitAtRealGaps(row, atoms, 0.8 * baseSize);
  return rows;
}
function pairRows(rows, baseSize) {   const candidates = [];   for (let i = 0; i < rows.length; i++) {     for (let j = i + 1; j < rows.length; j++) {       if (!rowBlocks(rows, i, j)) continue; // nothing between the rows       if (!rowsOverlap(rows[i], rows[j], 0.4)) continue;       candidates.push({ i, j, dy: rows[j].baseline - rows[i].baseline });     }   }   candidates.sort((a, b) => a.dy - b.dy); // tightest pair first   return candidates; }

The blocking check is the load-bearing one. A denominator's row must be vertically adjacent to its numerator's row. If a third full-size row sits between them, the pair is not a fraction, it is two separate lines, and treating it as one is how nested expressions get flattened into nonsense.

The axis of the whole expression

Once rows exist, the expression needs a baseline of its own, the math axis. A flat line of y = mx + b has one baseline. A fraction has two. A formula with a fraction and a trailing term has three.

The axis is the median of the row baselines, weighted by row population. The median survives asymmetry better than the mean: a tall fraction stack over a single trailing term does not drag the axis down the way an average would. Every row that is not a fraction pair is then placed relative to that axis, and superscripts and subscripts attach to their parent by horizontal reach, not by vertical distance.

// The axis is the median of row means, not the average
function mathAxis(rows) {
  const means = rows.map(r => r.meanBaseline).sort((a, b) => a - b);
  const mid = Math.floor(means.length / 2);
  return means.length % 2 ? means[mid] : (means[mid - 1] + means[mid]) / 2;
}

Gates must measure the population they govern

The most counterintuitive failure came from a size threshold. To decide which glyphs are "full size" rows and which are scripts, the algorithm compares each glyph against a body size computed from the expression. That threshold looked innocent, and it silently flattened every fraction in documents that contained a square root.

The radical glyph is tall. It stretched the body size estimate, the threshold rose, and suddenly no row qualified as a fraction row anymore. A lone sqrt anywhere in the paragraph starved the pairing rule for everything else.

The fix is a discipline, not a constant: compute the size baseline from the population the rule applies to. Stretch delimiters, radicals and tall brackets are excluded from the body size estimate, because they are exactly the glyphs whose height is structural, not typographic. Scripts are still gated by size, but the gate is no longer corrupted by the one glyph that is tall on purpose.

// Body size excludes stretch delimiters: radicals and tall brackets
const bodySize = median(
  atoms.filter(a => !isStretchDelimiter(a)).map(a => a.size)
);

Implementation evidence

The full algorithm ships in the extraction pipeline and renders with a vendored LaTeX renderer. The pipeline emits a paragraph class for math blocks, carries the reconstructed LaTeX in a data attribute, and renders the block to HTML at assembly time. The rendered markup is passed through the same sanitizer as every other document fragment, which required explicitly allowing inline style attributes.

Three regression suites cover it. One suite asserts exact LaTeX output for 25 expressions, including the quadratic formula, nested fractions and radicals with exponents. A second suite assembles full pages and asserts the math block appears with a rendered body and the flattened text does not survive anywhere outside the data attribute. A third suite pins the export stylesheet: fonts inlined as base64 data URIs, generated at build time so it can never drift from the vendored renderer.

Tradeoffs

The row model costs complexity. Per-atom pairing was thirty lines and wrong; row clustering is three hundred lines and right. The gap measurement against farthest absorbed edges is conservative, which means some genuinely separate expressions can be glued when a superscript hangs across a column break. That case is accepted because it is rare, and because the flattened alternative was wrong every time.

The accepted ambiguity is honest: two adjacent fractions with no operator between them, a/b c/d, collapse into a single fraction ac/bd. Mathematically equivalent, structurally different. The separated case with an operator between them is blocked correctly, and the test suite pins both behaviors.

The general lesson

When a pipeline flattens structure, do not rebuild the structure with the same locality the flattening destroyed. Recover the intermediate representation first: rows, axes, sizes. Every decision that follows is then made against a model of the page, not against a neighbor search.