The format won't label your figures: a postmortem
TLDR: We built image handling around the renderer's image events and assumed that covered "pictures." It covered raster XObjects only. Every vector-drawn figure, which is most technical figures, fell through into the table pipeline and came out as garbage grids. The lesson: enumerate what a format actually tags, not what your mental model of "an image" includes.
The assumption that seemed reasonable
PDF.js fires operator-list events for images. We hooked them, computed bounding boxes, rendered crops from a high-resolution canvas, and shipped image extraction. Test PDFs with photos and logos worked. The feature was marked done.
The unstated assumption: "images in the document" and "image operators in the stream" are the same set.
When it failed
A 76-page furnace installation manual. Dimensional drawings, wiring diagrams, isometric assembly figures on a third of the pages.
Extraction produced tables where the diagrams should be. One page turned a field wiring diagram plus a real electrical data table into a single bogus 13 by 9 lattice. Another page produced a 4 by 7 table containing zero text items, reconstructed entirely from a relay diagram's boxes. The real tables adjacent to these figures were swallowed or distorted.
Nothing crashed. Confidence scores were fine. The output was simply wrong in a way only a human comparing against the source would notice.
What was actually wrong
Technical figures in born-digital PDFs are almost never rasters. CAD exports, EDA tools, and Illustrator all emit vector paths: lines, beziers, polygons.
The renderer reports those as path construction operators, the same operators that draw table borders. Our pipeline had exactly two buckets for non-text ink: "image" (raster events) and "table evidence" (path segments). Vector figures had no bucket, so they landed in table evidence by default.
Measured on the failing pages: 2,900 to 6,500 path segments per page from figures alone, against under 200 segments for a genuinely bordered table. The table detector was not misbehaving. It was fed a haystack and asked to find hay.
What got deleted, what replaced it
Nothing was deleted, which is part of the lesson: the fix was a missing classifier, not a wrong one.
We added a figure detection pass that runs before table detection. It identifies strokes that came from multi-segment free paths with non-axis orientation (flattened curves, diagonals), clusters them spatially, and emits figure regions. Those regions remove their segments from the table pool, claim their text labels, and get cropped from the canvas render like any raster image.
The first version of the predicate ("short or diagonal segments") immediately painted a false figure over the numbers column of a balance sheet, because tables emit tiny fragments too. The stable version keys on subpath provenance, which turned out to be a clean separator: table pages emit zero multi-segment free-path strokes.
The generalizable lesson
When you build on a rendering library, write down the exhaustive list of what the format explicitly tags. For PDF that list is short: text runs, raster XObjects, annotations, and (rarely) structure tags. Everything else is untyped ink.
Then ask, for each content type your users will recognize ("figure", "chart", "table", "divider", "logo"), which tagged primitive it maps to. Any type that maps to "untyped ink" needs its own classifier, or it will be classified by whichever detector happens to run first.
The failure mode is silent because every detector downstream of the gap is working correctly on the wrong input. You will not find it in unit tests of the detectors. You find it by rendering the source page next to the output and looking.