Your manual override should send a diff, not a state
TLDR
When a user corrects an automatic system, send what they changed, not what they are looking at. Shipping the full state makes touching the UI a destructive operation, and it silently routes work that was already correct through your worst code path.
The position
A manual correction is a delta against a specific baseline. It is not a snapshot of the editor.
Any override API whose payload is "here are all the objects" is wrong, even when the objects are byte-identical to what the backend produced a moment earlier. The backend cannot tell the difference between "I moved this" and "this is exactly what you gave me", so it has to treat all of them as authoritative user intent.
What the industry does and why
Full-state payloads dominate correction UIs, and for decent reasons.
They are trivially idempotent. There is one code path, not two. The editor already has the array in memory, so passing it through is one line. And REST habits push toward PUT-the-resource over PATCH-the-change.
For editing a form or a document, that is fine. The user's state is the truth, and there is no competing producer.
Why it fails for this problem class
Correction UIs are different in one specific way: the backend is also a producer. The automatic system generated the objects. The user is amending a few of them.
That changes what the payload means. In a form, state and intent are the same thing. In a correction UI, state is mostly the machine's own output handed back to it, wrapped in a flag that says "trust this over your own judgment".
Three consequences follow.
Accidental destruction. In the case that prompted this, the editing canvas populated its working array on the first pointer event, because selection is by index and needs a stable array. Clicking anywhere and re-running then shipped all 75 regions as overrides. Since an override suppresses any detected region it overlaps, the page's real classification was deleted and replaced by the override rebuild, on an interaction that changed nothing.
The override path is always worse. It has to rebuild rich objects from a rectangle and a label. Ours guessed vertical centres from box midpoints instead of text extents and moved every region down the page. It reimplemented two structure detectors and disagreed with both. That is not sloppiness, it is inherent: a rectangle carries less information than a detection, and the gap has to be invented.
Bugs hide. Those defects existed for as long as the feature did. They were invisible because the path normally ran with one or two hand-drawn regions. The full-state payload turned it into the path for everything, and only then did the duplication and the flattened tables show up.
The better approach
Diff against the baseline you were given, and send only what moved.
Deleted, hand-drawn, retyped, moved, resized: send it. Untouched: leave it locked to the path that already handled it correctly. Compare positions with a small epsilon so a drag that lands back where it started is not counted as an edit.
The upside compounds. Untouched objects keep the better answer for free, and you rebuild one region instead of eighty, so the correct behaviour is also the fast one. After the change, editing a region and re-extracting produced output byte-identical to the untouched run, including when the edited region was the table itself.
What you give up
You need a stable identity per object and a baseline to compare against, which is real work if your detector does not emit ids.
You get two code paths where you had one, and they can drift. And the diff has an epsilon in it, which means a genuine sub-pixel adjustment is discarded as noise.
When the common pattern is right
When the user is the only producer. A drawing tool, a form, a document editor: there is no machine answer to preserve, so full state is the truth and a diff buys you nothing but complexity.
The moment a machine generates the objects and the human amends them, the payload has to say which is which.