We Shipped Sign-In. It Was Silently Failing.
TLDR
Our editor sign-in never completed for a single user, and we did not find out from monitoring. Our alerting counted errors, and this failure produced almost none, so the graph that would have exposed it was one we had never built: the count of people who actually got in.A metric that only counts failures cannot see a total failure
Most teams instrument the unhappy path first. You catch the exception, tag it, ship it to your error tracker, and put a threshold on it. That is the right instinct and it covers most of what goes wrong.
It does not cover this. Our sign-in flow handed work between two systems, and when the handoff broke, neither side raised an error it recognized as one. The first system finished its job and reported success, correctly, because its job really had finished. The second system waited for something that never arrived and eventually gave up, which it reported as a timeout. A timeout looks like a slow network. We had a small number of those every week already.
So the error rate barely moved. What moved was a number we were not looking at, because it had been zero since the day we launched and a flat line at zero draws no attention when nothing is plotting it.
The account count kept climbing the whole time. People signed up, tried to use the thing they signed up for, and quietly left. We spent real effort on onboarding copy trying to fix a funnel that was not leaking. It was severed.
Instrument the completion, not just the error
The fix to the measurement problem is smaller than the fix to the bug. Emit an event when the flow completes, and alert when its rate hits zero.
// Not enough on its own. This fires only when something throws,
// and a broken handoff between two systems often throws nothing.
track('sign_in_failed', { stage });
// The one that matters. A flat zero here is an outage, // whether or not anything errored. track('sign_in_succeeded');
Alerting on the absence of a success is different from alerting on the presence of an error, and the difference is exactly the class of bug that hides between two services. Anywhere a flow crosses a boundary, one side can be entirely healthy while the flow is dead.
Here is the query we should have been running from day one, and were not:
event stage count()
sign_in_started - 5
sign_in_failed poll_http 5Five attempts, five failures, zero completions. That took about ten seconds to run once we thought to ask the question. It would have returned the same shape at any point in the previous launch.
A value copied into two systems will drift, and nothing will tell you
The underlying bug was mundane. Two independent platforms needed to agree on one configuration value. One of them managed that value automatically and kept it current. On the other, it had been entered by hand and stayed frozen at whatever it was.
For a while they agreed. Then the managed side rotated, which is what managed sides are supposed to do, and the two halves stopped matching. Nothing in either system considered this a problem worth mentioning, because from each platform's point of view the value was perfectly valid. Only the relationship between them was broken, and no single system owns a relationship.
This is worth generalizing. Any value duplicated across two deployment systems will eventually diverge, and neither system will warn you, because neither one knows the other exists. If two services must agree on something, either derive it from one source both of them read at runtime, or check the agreement explicitly and fail loudly when it breaks. Copying a string into two dashboards and trusting it to stay put is not configuration. It is a bet on nobody ever rotating anything.
Do not announce a fix you have not counted
The part that stings.
We had already published this in a release note:
Fix: sign-in never worked.
In a previous version we published this:
Correction. That release said sign-in was fixed. It was not.
Both are real entries in our own changelog, and they are still there, because deleting the first one would be worse.
What happened in between is that we found a genuine bug on the sign-in path, fixed it, watched sign-in work on a development machine, and shipped the note. There was a second bug behind the first. Our own developer setup did not reproduce it, since a local environment reads its configuration from one place instead of two, which is precisely the condition that made the bug impossible.
The mistake was not missing the second bug. Two bugs on one path is ordinary. The mistake was writing "fixed" on the strength of one machine working once, when the number that would have settled it was a single query away and we did not run it.
A fix is not confirmed by the absence of the symptom on your laptop. It is confirmed by the success metric moving in production. If that metric does not exist yet, building it is part of the fix, not follow-up work.
What we changed
Beyond the configuration itself, three things:
- The completion event now exists and is alerted on. A zero rate pages someone.
- The waiting side distinguishes a problem that retrying can solve from one it cannot, and says so instead of reporting every failure as a generic timeout.
- Release notes claiming a fix on a measurable path have to cite the measurement. If the number is not in the note, the note does not go out.
Rule of thumb: if a flow crosses a boundary between two systems, count the completions. Errors tell you something broke; only completions tell you the whole thing still works.