We shipped a rule engine that had never once fired

TLDR: We built an electrical-rule checker, confirmed it ran, and moved on. An audit later found it had produced zero findings on genuinely broken circuits the entire time. The rules executed. They just looked up pin roles by an identifier the data never used, so every rule concluded there was nothing wrong.

The assumption that seemed reasonable

We had a validation layer with a spec table and a rule pack. The spec said what each component's pins do: this pin is an input, that one is an output, this one is power. The rules read those roles and flagged bad wiring: two outputs shorted together, a net with no driver, power shorted to a signal.

We wrote the spec. We wrote the rules. We ran a syntax check. We drew a circuit and it rendered. We wrote it up as done.

The assumption underneath all of that: if the validation code runs without error on real input, it is validating. That assumption feels so obviously true that nobody states it out loud. It is also, in this case, completely false.

When it failed

It did not fail with an error. It failed with silence, and silence does not page anyone.

The signal came from a human. Circuits generated in the tool kept coming out electrically wrong, outputs tied to outputs, parts wired backwards, and the design checker said nothing was wrong. If the checker was working, either the circuits were fine or it should be screaming. It was doing neither.

So we drew the most obvious possible violation by hand: two identical gate outputs into the same node. That is a textbook short. We ran the check. It returned the topological findings it always returned, an unconnected pin here and there, and not one word about the short.

That was the moment the assumption died. The rule for that exact violation existed in the codebase. It had been "running" for a whole development cycle. It had never fired.

What was actually wrong

The rule looked up each pin's role from the spec table using the pin's identifier. The spec was keyed by human-readable names: IN1, IN2, OUT, Anode, Base, Gate. But the pins in the actual graph carried the identifiers the rendering kit had stamped on them: in1, in2, out, 0, 1, base, gate.

Different case. Different scheme entirely for two-terminal parts (0 and 1 versus A and B). One spec even used a Unicode minus sign where the kit used an ASCII hyphen.

Every lookup returned undefined. The rule collected the roles of the pins on a net, got a bag of nothing, and correctly concluded that a bag of nothing contains no two outputs. Correct logic, empty input, useless result.

Two independently authored sources, the spec and the kit, were joined on an identifier, and nobody had ever checked that the identifiers matched. They never had.

What got deleted

Nothing dramatic got deleted. The rules were fine. The bug was in the keys, and the fix was to re-key the spec to the identifiers the data actually uses. We pulled those identifiers straight out of the live kit rather than guessing again, and every guess we had originally made turned out wrong.

The negative test that followed deleted something more valuable: a second bug we had not noticed. Once the rules could see roles, a correct circuit started throwing false alarms. We had modeled diode and LED pins as input/output to sneak in polarity, and that made passive parts look like signal drivers. A cathode wired to ground read as a short. We had to delete the input/output typing on those parts and model polarity as its own property. That bug had been hiding behind the first one. You cannot see your false positives while your true positives are dead.

What replaced it

Three things replaced the old "it runs, we're done" workflow.

A comment above the spec that lists every real identifier the kit emits, so the invariant is visible to the next author instead of living in nobody's memory.

A rule that reads identifiers from the runtime source of truth at authoring time, not from what we assume they are called.

And the test we should have written first: assert that a known violation produces a finding. Draw the short, expect the error. If that test had existed, the mismatch would have failed CI immediately and this postmortem would not exist.

The generalizable lesson

Any validation layer that joins two data sources on an identifier has a hidden dependency: the identifiers must match, and nothing enforces that by default. A miss returns undefined, undefined reads as "nothing to check," and "nothing to check" reads as "all clear."

So the health of a validator is never proven by it running. It is proven by a known-bad input producing a known finding. Until you have that test, "the checker passes" and "the checker is dead" are indistinguishable, and you will pick the reading that lets you move on.