Security Checklists Are Better Than Security Intuition

TLDR: We found 7 real auth issues using a checklist. Code review had missed all of them. This is not a coincidence. It reflectsis a structural property of how bugs form in security-sensitive code.

The Problem With Relying on Intuition

Security code review is tough because the bugs are often hidden. For example, missing the frame-ancestors 'none' CSP directive does not cause a crash. Using a wildcard '*' in a postMessage call works fine during tests. An error message like "Email not found" instead of "Invalid email or password" will passes all functional tests. These issues do not cause errors but, but they leave vulnerabilities.

Experienced engineers miss these issues constantly. Not because they are careless, but because intuition-based code review focuses onis optimized for finding functional bugs. Security bugs are non-functional defects. They do not cause incorrect output but. They permit incorrect input.

Checklists Encode Accumulated Incidents

Every item on a good security checklist corresponds to a real incident or class of incidents. The OWASP ASVS, NIST 800-63B, RFC 6819 were written by people who observed real breaches and extracted structural patterns.

The 2013 Facebook OAuth open redirect (4.3 in our checklist). The 2020 GitHub URL-logging incident (1.2). The alg: none JWT attacks (7.2). These are not hypothetical attacks. They happened in production at scale. The checklist exists so you do not have to rediscover them yourself.

When you audit by intuition, you redo the work of everyone who has audited auth code. When you audit with a checklist, you get the accumulated output of every prior security incident.

The Backup-Path Problem

Our most interesting finding was not the most severe. The poller_secret in a URL query param was the critical finding, but it occurred in a backup proxy file, not the primary implementation.

The primary file was correct. The backup file was a draft from an earlier iteration. The security review focused on the primary path and did not check all implementations of the same pattern.

This class of bug is almost never caught by intuition-based review. You review the code you wrote but not the code you forgot. A checklist saying "credentials must not appear in URLs" makes you search every call site, not just the one you remember.

Line-by-Line Review vs. Requirement-by-Requirement Audit

The difference is the unit of analysis. Line-by-line review asks: "Is this line correct?" Requirement-by-requirement audit asks: "Does the codebase satisfy this invariant everywhere?"

"Credentials must not appear in URLs" is a global invariant. You cannot satisfy it by reviewing one file. You must grep the entire codebase for every URL construction and verify none contain credentials. That is what the checklist forced us to do and how we found the secondary proxy file.

The Industry's Comfortable Lie

Most teams do a security review before each major release. They read changed files, maybe run a scanner, and call it done. This is mostly theater. Changed files are reviewed; unchanged files are not. The assumption is that old code is known good. But old code is just code not audited recently.

The backup proxy file with the bug was not new code. It was written months ago, reviewed then for functional correctness, and forgotten. It was not part of any recent diff and would never appear in a "review the PR" workflow. Only a checklist-driven search of the whole codebase found it.

What This Costs

About half a day: reading the checklist, writing the audit table, grepping each requirement. Not a small time investment for a solo project but smaller than the cost of a credential appearing in production logs for months.

The uncomfortable math: the expected cost of a credential exposure is much higher than the cost of an audit, even at low probability. The checklist is cheap insurance with a guaranteed output: a complete list of what you checked, what passed, and what you fixed. No intuition-based review produces that output.

The Takeaway

Run the checklist. All 27 items. Every time the auth surface changes significantly. Keep the audit table in the repo. It is documentation that the check was done, not just a record of the findings.

The findings will surprise you. They surprised us.