Your extractor should never merge text it can link instead

TLDR: When an extractor decides two text fragments are one paragraph, the standard move is to concatenate them and emit the merged result. That throws away the spatial truth to serve the semantic one. Emit both fragments with a link between them and let each consumer pick its truth.

The position

An extraction pipeline should never destroy information at a decision point it might be wrong about. Fragment relationships are decisions, not facts. Record them as links and defer the join to the consumer.

What the industry does

Nearly every PDF-to-text tool normalizes destructively. Column fragments get concatenated, hyphens get stripped, reading order gets baked in, and the output is one linear text stream.

The reasoning is sensible: most consumers want clean text, and a single output format is simpler than a graph. If your only product is a .txt file, merging at extraction time looks free.

Why it fails for this problem class

It fails the moment you have two consumers with different truths. A design-faithful view needs the fragment where the source painted it. A semantic view needs the sentence whole. A merge can satisfy exactly one of them.

It also converts heuristic errors into permanent corruption. Continuation detection is probabilistic: a wrongly merged heading is now fused into a paragraph, and no downstream consumer can recover the original because the original is gone. With a link, the same error is one bad edge that any consumer can ignore, filter, or let a user delete.

And it forecloses the most interesting output: responsive documents. If the goal is a page that renders the original design at full width and reflows like a website below it, you need the fragments AND the flow, simultaneously, in one artifact.

The better approach

Attributes, not concatenation:

<div id="f7" data-flow-next="f8">The day is</div>
<div id="f8" data-flow-prev="f7" data-continuation>brighter.</div>

The spatial renderer ignores the links. The text exporter walks them and joins with the recorded mode (space, dehyphenate, keep-hyphen). The responsive stylesheet fuses continuations only when columns physically stack. One extraction, several truths, nothing deleted.

The join logic runs once at extraction time either way. The only difference is whether its output overwrites the fragments or annotates them.

What you give up

Size and simplicity. Every consumer must understand the link vocabulary, and a naive consumer that ignores it gets fragmented text (exactly what merging tools output when they guess wrong, so the floor is the same).

You also give up the comfort of a single canonical string. If a downstream system demands one, you materialize it by walking the chain, which is ten lines of code.

When merging is right

If your pipeline has exactly one consumer, forever, and it wants linear text (a search indexer, an LLM ingestion job), merge away. The dual-truth machinery earns nothing there.

Merge is a rendering, not a representation. Render at the edge, keep the graph in the middle.