Security Checklists Are Better Than Security Intuition

TLDR

We recently ran a structured 27-point security checklist audit across our auth surfaces and discovered 7 genuine vulnerabilities that passed every standard pull request code review. Intuition-based reviews are optimized for catching functional bugs, but security vulnerabilities are non-functional defects. They do not throw errors, they silently allow invalid inputs or log sensitive credentials.

Audit Approach	Primary Focus	Defect Type Caught	Catch Rate on Legacy/Secondary Code
Intuition-Based PR Review	Line-by-line function correctness	Functional crashes & syntax errors	Low (skips un-diffed legacy files)
Checklist-Driven Requirement Audit	Repository-wide invariant checks	Non-functional security flaws	High (greps every call site across repo)

---

Intuitive code reviews fail to catch non-functional security flaws

When developers review security-sensitive code, we rely on intuition. We read pull request diffs, test happy-path sign-ins, and check for obvious exceptions. But non-functional security flaws do not break execution:

• A missing frame-ancestors 'none' CSP header will not crash your web app.

Passing as the target origin in window.parent.postMessage() works fine in local development tests.

• Returning "User not found" instead of "Invalid credentials" passes every functional test suite while opening the door to user enumeration.

Because these defects never trigger runtime errors, intuition-driven code reviews miss them almost every time.

---

Isolated PR diffs obscure un-modified legacy vulnerabilities

In our codebase, the most critical finding, a bearer polling secret leaking in a URL query parameter, sat inside a secondary backup proxy file written months earlier.

When we originally hardened our primary authentication proxy, the PR diff only contained the primary file. The code review passed because the primary file looked spotless. But because nobody grepped the entire codebase against an invariant requirement ("credentials must never appear in URLs"), the legacy backup proxy was left untouched in production.

Line-by-line PR reviews ask whether the modified lines look correct. Requirement-driven audits ask whether the entire codebase satisfies the security rule everywhere.

Intuition PR Review (Diff-Only):
[Changed Code Only] ---> Checked ---> Passed (Misses un-modified files)
         ^
         |
    [Legacy Code] (Leaking parameter ?secret=Y remains in production)
Checklist Repository Sweep (Full Repo): [OWASP Invariant Check] ---> Scan entire codebase ---> Exposes legacy leaks                                                              |                                                              v                                                      [Fully Remediated]

---

Structured repository-wide checklists expose hidden vulnerabilities

To eliminate these blind spots, we adopted a mandatory 27-point security checklist derived from OWASP ASVS, NIST SP 800-63B, and RFC 6819. Instead of auditing files in isolation, we audit by requirement across all files:

1. Global invariant grepping: Instead of reading individual files, we execute repository-wide searches for vulnerable patterns (such as searching for searchParams.set or postMessage(*)).
2. Auditing the unchanged code: Legacy proxies and shared utility components are audited alongside new code.
3. Persisting the audit matrix: We store the audit table directly inside the repository as living documentation of checked invariants.

Rule of thumb: Never limit security audits to git diffs. When hardening a security-sensitive pattern, grep the entire codebase for every implementation of that pattern before marking the task done. Fixing the main path is not the same as fixing all paths.