A manual override must reproduce the pipeline, not replace it

TLDR

If you let users hand-correct the output of an automatic classifier, the override path will grow into a second implementation of your pipeline. That second implementation is always worse, because it only knows the shape the user drew. The fix is to make an override a selector over the real pipeline's work rather than a replacement for it, and to send only the things the user actually changed.

The problem class

You have a detector that finds structure in messy input. Document regions, audio segments, bounding boxes in an image, spans in a log. It is right most of the time and wrong occasionally, so you add a manual correction surface: draw a box, pick a type, re-run.

The correction surface has a payload, usually a rectangle and a label. The backend has to turn that rectangle back into the same rich object the detector produces. That is where it goes wrong.

The naive approach

Treat the user's rectangle as a complete specification. Collect everything inside it, build the structure from what you collected, emit a region.

for (const custom of overrides) {
    const items = allItems.filter(it => inside(it, custom.bbox));
    const grid  = buildGridFromItems(items);
    out.push({ type: custom.type, bbox: custom.bbox, items, grid });
}

This reads as obviously correct and it is obviously wrong in three separate ways.

Failure one: the blast radius

The correction UI needs a stable, mutable array to hit-test clicks against, so the first click on the canvas copies every detected region into the editable set. That copy is a UI concern.

It stops being a UI concern the moment the whole array is shipped as the override payload. Downstream, an override suppresses any naturally detected region it overlaps by more than 40%. Sending all of them therefore replaces the entire page's real classification with the crude rebuild, on a click that changed nothing.

The fix is a diff. Compare each editable region against the baseline it was copied from, and send only the ones that differ.

const EPS = 0.5;   // sub-pixel drift from the canvas/worker scale round trip
const edited = editable.filter(r => {
    if (r.deleted || !r.id) return true;          // deleted, or hand-drawn
    const base = baseById.get(r.id);
    if (!base) return true;                       // duplicated or merged
    if (r.type !== base.type) return true;        // retyped
    return Math.abs(r.bbox.x - base.bbox.x) > EPS
        || Math.abs(r.bbox.y - base.bbox.y) > EPS
        || Math.abs(r.bbox.w - base.bbox.w) > EPS
        || Math.abs(r.bbox.h - base.bbox.h) > EPS;
});

Note the epsilon. A drag that lands back where it started stamped an "edited" marker on the way past, but it is not an edit. Everything unedited stays locked to the normal path, which is also strictly less work.

Failure two: the rebuild does not know what the detector knew

An override carries a rectangle. The detector's output carries a rectangle and everything it inferred: font size, the vertical centre it measured off the text rather than the box, its column assignment, its reconstructed grid.

The transport between the UI and the engine had stripped most of that, so the rebuild had to invent it. Every substitution it made was a regression.

The vertical centre is the clearest example. The pipeline measures a text region's centre from the text extent; the bounding box is padded by a line height, so its midpoint sits several pixels lower. Using the box midpoint moved every overridden region down the page.

// Text-flow types measure their centre off the glyphs. Images and boxes
// genuinely use the box midpoint, so measuring their label text instead
// would move them relative to the natural result.
yCenter: (TEXT_FLOW_TYPES.has(type) ? textYCenter(matchedItems) : null)
      ?? (bbox.y + bbox.h / 2)

The rule that falls out: for every property the rebuild has to supply, find out how the real pipeline derives it and derive it the same way. Not approximately. Identically.

Failure three: the rebuild is a second, worse algorithm

This is the expensive one.

Both of the structure builders in the override path were reimplementations. One read a grid off ruled lines inside the rectangle. One banded text into rows and columns. Both looked reasonable and neither agreed with the detectors they were standing in for.

The ruled one failed because it fed the reconstructor a subset of the page's line segments, the ones near the user's box. The reconstructor derives its row and column clustering thresholds from the extent of the segments it is given. On a real architecture table, the same 24 segments reconstructed to a 10 by 6 grid inside the full page set and to a 2 by 4 grid on their own. A nine row table came back as its header strip.

The banded one failed more quietly. It grouped rows by vertical position alone, while the real detector groups bands by an adaptive gap. Eight rows became sixteen. Same text, wrong shape.

The correct structure is not "reimplement the detector against a bbox". It is:

// Reconstruct over the WHOLE page, exactly as the detector does, then let
// the region pick the candidate it overlaps most.
const pageGrids = reconstructAll(allSegments, opts);
const grid = pickByOverlap(pageGrids, custom.bbox);

and, for the borderless case, calling the real detector on the claimed items and picking the best candidate, with the hand-rolled bander demoted to a fallback for when the detector's gates reject a box the user insists is a table.

The model that works

An override is a selector plus a delta, never a replacement pipeline.

• Selector: run the real detectors over the whole input, then let the user's rectangle choose which of their results applies. The rectangle answers "which one", not "what is it".
• Delta: send only what changed. Everything untouched keeps its original, better answer for free.
• Fallback: keep a crude builder for the case where no real detector produced a candidate, because the user is allowed to be right where the detector found nothing.

Tradeoffs

Running the full detector to serve one override costs more than filtering a rectangle. On a page it is a few milliseconds, and the diff means you usually run it for one region instead of eighty, so the net is faster.

The bigger cost is coupling. The override path now depends on detector internals that were previously private, and a change to a clustering threshold will move override output with it. That is the correct coupling. The alternative was two implementations drifting apart silently, which is exactly what happened.

The measurable result: with the fixes in place, editing a region and re-running produces output byte-identical to the untouched extraction on every page tested, including the case where the edited region is the table.