The br is not your friend in extracted PDF text

TLDR: A PDF extraction pipeline's sentence-aware paragraph joiner used <br> to keep visual line breaks inside the same <p> tag when a sentence spanned multiple lines. This preserved the page's visual structure but broke the text for every downstream use: editors showed fragments like "train-\n ing error" with visible hyphens, plain-text export had hard-wrapped lines, and reflow in responsive layouts was impossible. Replacing <br> with a space produced clean, reflowable text that matched what readers and editors expect.

The assumption that seemed reasonable

PDF text is extracted as items with exact x,y coordinates. A paragraph that spans multiple visual lines produces several items: "Unexpectedly,", "such degradation is not caused", "by overfitting, and adding", "more layers to a suitably deep". Each item has its own position and the gap between them indicates they are on separate lines.

The extraction pipeline groups these items into a paragraph region based on vertical proximity. Then it detects sentence boundaries: if the previous line does not end with terminal punctuation (. ! ? :) and the next line is not a list marker, the two lines belong to the same semantic sentence even though they are on different visual lines.

The original code joined these sentence fragments with <br>:

prev.inner += '<br>' + b.inner;

The reasoning was sound: <br> preserves the visual line break while keeping the content inside one <p> tag. The HTML is valid. The page looks the same as the PDF. Fidelity is maintained.

When it failed

The first sign of trouble was a paper abstract that came out looking like this:

<p>Unexpectedly,<br>such degradation is not caused by overfitting, and adding<br>more layers to a suitably deep model leads to higher train-<br>ing error, as reported in [11, 42] and thoroughly verified by<br>our experiments.</p>

That is a single sentence fragmented into five <br>-separated pieces. The hyphenated "train-\ning error" was particularly bad: the PDF's line-break hyphen became a visible artifact in the HTML. In any responsive layout, the <br> breaks destroyed reflow. In plain-text export, the <br> tags were invisible, producing run-together words. In the contenteditable editor, each <br> was a DOM node that the cursor had to navigate around.

The assumption that "preserving visual breaks maintains fidelity" was wrong for the wrong reason. The breaks are visual artifacts of the PDF's page layout, not semantic features of the text. A PDF page has a fixed width. The text wraps at that width. When you extract the text and put it in a variable-width container, those line breaks are meaningless.

What was actually wrong

The pipeline was conflating two definitions of "paragraph":

1. The visual paragraph: text items grouped by vertical proximity on the PDF page
2. The semantic paragraph: a complete thought bounded by sentence-ending punctuation

The <br> logic was trying to preserve visual line structure within semantic paragraphs. But for extracted PDF text, visual line structure is an artifact of the source page width, not a property of the text. Preserving it created problems for every consumer of the output.

The hyphens compound the problem. PDF typesetting engines hyphenate words at line breaks. The hyphen character is part of the extracted text string. When a hyphenated break is joined with <br>, the reader sees "train-" on one line and "ing error" on the next, which looks like a typo.

What got replaced

The change was three characters: '<br>' became ' '.

prev.inner += ' ' + b.inner;

The same sentence now produces:

<p>Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher train- ing error, as reported in [11, 42] and thoroughly verified by our experiments.</p>

The remaining "train- ing" artifact is a separate problem (the PDF's hyphenation character is in the extracted text and needs a dehyphenation pass that knows the word boundary). But the <br> no longer makes it worse. The text reflows naturally in any container. The plain-text export is clean. The contenteditable editor has no extraneous DOM nodes.

The generalizable lesson

When transforming extracted data for human consumption, distinguish between structural fidelity (which elements exist, their order, their hierarchy) and presentational fidelity (how they were arranged on the source medium). Preserving presentational fidelity at the expense of structural cleanliness creates artifacts that hurt every downstream use. A <br> in extracted PDF text is not a semantic line break. It is a fossil of a typesetting engine's line-wrap algorithm. Fossilize it at your own risk.