Hot Take: Security Checklists Are Better Than Security Intuition

TLDR: We found 7 real auth issues using a checklist. Code review had missed all of them. This is not a coincidence — it is a structural property of how bugs form in security-sensitive code.

The Problem With Relying on Intuition

Security code review is hard because the bugs are not obvious. A missing frame-ancestors 'none' CSP directive does not crash anything. A wildcard '*' in a postMessage call works fine in testing. An error message that says "Email not found" instead of "Invalid email or password" passes every functional test. None of these things produce an error. They just leave a door open.

Experienced engineers miss these constantly. Not because they are careless, but because intuition-based code review is optimized for finding functional bugs. Security bugs are non-functional defects. They do not cause incorrect output — they permit incorrect input.

Checklists Encode Accumulated Incidents

Every item on a good security checklist corresponds to a real incident or a class of real incidents. The OWASP ASVS, NIST 800-63B, RFC 6819 — these documents were written by people who watched real breaches happen and extracted the structural patterns.

The 2013 Facebook OAuth open redirect (4.3 in our checklist). The 2020 GitHub URL-logging incident (1.2). The alg: none JWT attacks (7.2). These are not hypothetical attacks. They happened in production at scale. The checklist exists so you do not have to rediscover them yourself.

When you audit by intuition, you are redoing the work of everyone who has ever audited auth code. When you audit with a checklist, you get the accumulated output of every security incident that came before you.

The Backup-Path Problem

Our most interesting finding was not the most severe. The poller_secret in a URL query param was the Critical finding, but it happened on a backup proxy file — not the primary implementation.

The primary file was correct. The backup file was a draft copy from an earlier iteration. The security review at the time focused on the primary path and did not enumerate all implementations of the same pattern.

This is a class of bug that intuition-based review almost never catches. You review the code you wrote. You do not review the code you forgot about. A checklist that says "credentials must not appear in URLs" makes you search for every call site, not just the one you remember writing.

Line-by-Line Review vs. Requirement-by-Requirement Audit

The difference is the unit of analysis. Line-by-line review asks: "Is this line correct?" Requirement-by-requirement audit asks: "Does the codebase satisfy this invariant — everywhere?"

"Credentials must not appear in URLs" is a global invariant. You cannot satisfy it by reviewing one file. You have to grep the entire codebase for every URL construction and verify none of them contain credentials. That is what the checklist forced us to do, and that is how we found the secondary proxy file.

The Industry's Comfortable Lie

Most teams do a security review before each major release. They read through the changed files, maybe run a scanner, and call it done. This is almost entirely theater. Changed files are reviewed; unchanged files are not. The assumption is that old code is known-good. But old code is just old code that has not been audited recently.

The backup proxy file that had the bug was not new code. It was written months ago, reviewed at the time for functional correctness, and forgotten. It was not part of any recent diff. It would never have appeared in a "review the PR" workflow. Only a checklist-driven search of the whole codebase found it.

What This Costs

About half a day. Reading the checklist, writing the audit table, grepping each requirement. Not a small time investment for a solo project. But smaller than the cost of a credential appearing in production logs for months.

The uncomfortable math: the expected cost of a credential exposure is much higher than the cost of an audit. Even at low probability. The checklist is cheap insurance with a guaranteed output: a complete list of what you checked, what passed, and what you fixed. No intuition-based review produces that output.

The Takeaway

Run the checklist. All 27 items. Every time the auth surface changes significantly. Keep the audit table in the repo. It is documentation that the check was done, not just a record of the findings.

The findings will surprise you. They surprised us.