Green Tests and the Unverified System: Why Coverage Belongs in the Verdict

TLDR: A check that does not report how much it examined is not a check, it is a mood. I had 519 passing tests across eleven suites while a core feature had never worked a single time in production. The tests were not wrong. They were answering a narrower question than the one I believed they answered, and nothing in a green run tells you which question that was.

This is the pattern behind four separate bugs I found in one project over two weeks. Each looked different. Each was the same mistake at a different altitude.

The first verdict that lied

I built an electrical rule checker. Twelve rules: unconnected pins, shorted power, conflicting drivers, duplicate designators. Standard stuff. It ran on a schematic and returned findings.

Then I fed it a circuit that destroys itself on the first switch-off. A motor driven by a transistor with no flyback diode. The collapsing magnetic field puts hundreds of volts across the switch and kills it.

The checker returned clean. No errors, no warnings.

It was not broken. Every one of those twelve rules is a graph property: is this pin connected, do these two outputs collide. None of them reads a component value, and none of them knows what a component is. A motor with no flyback diode is a perfectly legal graph. It is only wrong if you know that a motor is an inductive load, and that is domain knowledge, not topology.

The failure was not the missing rule. The failure was that "clean" and "I checked nothing relevant" produced identical output.

What a verdict has to carry

The fix was not just adding rules. It was making every verdict-producing layer report its own coverage.

{
  findings: [],
  coverage:          { total: 14, verified: 14, unverifiable: 0 },
  knowledgeCoverage: { inductiveLoads: 1, inductiveLoadsChecked: 0 },
  busCoverage:       { i2cDevices: 0, i2cChecked: 0 }
}

Read that second line. One inductive load found, zero checked. The design has a relay in it whose coil is two of its five pins, and the rule that checks for flyback diodes only handles two-terminal loads. Which two pins are the coil is not derivable from the netlist.

So it does not guess. It does not flag. It reports that it found a candidate it could not examine.

That single number is the difference between "your circuit is fine" and "your circuit has one thing in it I am not equipped to judge." The first is a claim. The second is a fact.

The permissive default that silences a checker

Here is how the same class of bug appears one layer down.

The system lets a model define a custom part: a microcontroller, a sensor breakout, whatever. Each pin gets a role, so the rules can reason about it. Roles are things like power, ground, input, output.

The original code defaulted any pin with no declared role to passive.

That looks harmless. passive is the safe, neutral option, the one that will not cause false alarms. And that is exactly the problem. Eight of the twelve rules only fire on non-passive pins. A model defining a twenty-pin board without roles produced a schematic where eight rules went quiet and the final verdict read "wire correct: true."

The verdict was not lying about what it found. It was lying about what it looked at.

The fix was a seventh role, unspecified, which is explicitly not a synonym for passive:

// 'unspecified' means the pin exists and is drawn, but nothing declared
// what it DOES. It is counted by coverage() and must keep a verdict from
// claiming correctness.
const ROLES = ['passive', 'input', 'output', 'bidir', 'power', 'ground', 'unspecified'];

An unroled pin is now drawn, counted, and reported. It cannot be refused, because refusing costs the user their drawing. It cannot be promoted, because promoting is what made the verdict lie.

When you pick a default for missing data, you are choosing which failure mode you get. A default that suppresses checks is worse than one that produces noise, because noise is visible.

Then the tests themselves lied

By this point I had eleven test suites and 519 passing checks. Units parsing, layout planning, rule packs, symbol generation, contract validation. All green, all fast, all meaningful.

Then I put a real browser on the system for the first time and watched two tools try to send each other data.

Nothing arrived. Not a corrupted payload, not an error. A ten second timeout and silence.

The application is a host page with sandboxed tool frames inside it, exchanging messages. A frame asks the host to store a blob, gets back a pointer, and hands that pointer to another frame. Standard request and reply over a message channel.

The handshake had a deadlock in it, and it was deterministic rather than intermittent:

1. A tool announces itself as soon as its own document is ready
2. The host registers that tool when the frame's load event fires
3. A frame's document is ready before its load event. Always. Measured at 63ms earlier

So the announcement arrived while the host's registry was still empty. The host could not match the message to a known tool, so it never replied with the tool's assigned identity. The tool kept its identity as null forever.

Every subsequent message it sent was stamped with an unknown sender. Every reply the host tried to route looked up that unknown sender, missed, and dropped the response on the floor.

No error was logged anywhere. The only symptom was that every request timed out, and requests that time out after ten seconds in a background path are requests nobody notices failing.

Cross-tool transfer had never worked. Not once. Under 519 green tests.

Why no unit test could have caught it

This is the part worth taking away, and it is not "write more tests."

Every one of those 519 checks runs in Node. They can prove that a table contract validates a merged cell correctly, and they did. They can prove the sender constructs a valid payload, and they did. They can prove the receiver parses that payload correctly, and they did.

None of them can prove the message arrived, because there is no message. There is no host, no frame, no channel. The wire does not exist in that runtime.

The bug lived precisely in the gap between two components that were both individually correct.

sender    ✓ tested, correct
   ↓
[ the wire ]   ← nothing tested this
   ↓
receiver  ✓ tested, correct

I added a browser-level suite. Nineteen checks, about two minutes, driving the real application: real frames, real messages, real storage. It found the deadlock in the first run.

Then it found a second bug that had been sitting behind the first one the whole time.

The second bug, revealed only after fixing the first

There is a feature that pushes edited values from the table tool back onto the schematic. It shows a diff, splits changes into safe and needs-review, and applies the safe ones.

With the transport fixed, I ran it. The review dialog appeared. It correctly identified a value change from 10kΩ to 22kΩ, classified it as safe, and offered to apply it.

I clicked apply. The dialog closed. The canvas did not change.

// The lookup
const el = root.querySelector(g#${CSS.escape(change.id)}[data-symbol]);

change.id came from the exported netlist, which uses internal identifiers like component_0. The actual DOM element carries a generated id like sym_resistor_1786760018341. Two different namespaces, one field name.

The selector never matched. applied stayed at zero. The dialog closed regardless, because closing was not conditional on anything having happened.

The fix was to resolve through the component collection the rest of the editor already uses, rather than changing what the netlist exports, since three other consumers depend on that shape.

When a value crosses a serialization boundary and comes back as a lookup key, check that the key is the same namespace on both sides. A field named id is not automatically the id the other side means.

The three layers, and what each one cannot see

What I ended up with is not more tests. It is tests at three altitudes, with an explicit statement of what each is blind to.

Layer	Runtime	Catches	Cannot catch
Unit	~10s	Logic, parsing, contracts, rules	Anything on a wire or in a DOM
Protocol	~40s	Tool shape, marshalling, typed errors	Whether the UI works
Browser	~2min	Handshakes, real messages, real mutation	Auth, which is stubbed to reach the layer under test

That third column is the one that matters, and it is the one nobody writes down.

A test suite that does not state its blind spots gets read as if it has none. Mine was read that way for two weeks, by me, while a feature that had never worked sat behind it.

Make the regression fail on purpose

One more habit, cheap and worth more than it costs.

A regression test you have never seen fail is a test you cannot trust. After fixing both bugs, I reverted each fix individually and confirmed the suite caught it, with the right message and only the right message:

• Revert the handshake fix, the transport check fails first and everything downstream cascades
• Revert the id lookup fix, only the apply check fails

If reverting a fix does not turn the suite red, the test is decorative. That takes two minutes to verify and it is the only thing that tells you the test is load-bearing.

What this actually generalises to

Four bugs, one shape. A component reported success over material it never examined.

• The rule checker passed a circuit it had no rules for
• The role default silenced eight checks and still reported correct
• The unit suite proved both endpoints while the channel between them was dead
• The apply step closed its dialog whether or not it changed anything

The common fix is not more coverage in the test-percentage sense. It is making the report include the denominator.

"no problems found"                          ← unfalsifiable
"no problems found in 14 of 14 pins"          ← a claim you can check
"no problems found in 0 of 1 inductive loads" ← an honest admission

If your verifier cannot produce the third sentence, it cannot produce the second one either, and you have been reading the first one as if it were.

Tradeoffs, honestly

The browser suite is slow, needs a build step, and is more fragile than the unit tests. It stubs authentication to reach the layer under test, which means a green run says nothing about whether auth works, and that limitation is written into the file so nobody quotes a pass as evidence of something it never checked.

Coverage objects make results wordier. A caller has to decide what to do with "one candidate, zero checked," and the honest answer is usually to surface it rather than resolve it.

Both costs are real. Both are smaller than shipping a feature that has never worked behind a green suite.