The Bugs That Pass Every Check

Quick answer: every item on a security checklist is a presence check. Is the header set, is the key off the client, does the endpoint require auth. Those are mechanically detectable, which is why tools find them and why you have probably fixed them already. What remains is a harder class: code where each check passes and the composition is still wrong. An endpoint that verifies identity but never ownership. Two requests that are individually correct and corrupt a row when they interleave. Nothing is missing, so nothing flags it.

That distinction is the whole point. A missing header is an absence, and absence is detectable. An insecure direct object reference is a present check making the wrong comparison. It reads fine in review, passes its tests, and hands one customer another customer's invoice.

AI-assisted development produces this category at a higher rate, for a structural reason rather than a quality one. A model generates each endpoint as a self-contained unit that satisfies its own description. Correctness in a real system is mostly a property of how units interact under concurrency and partial failure, and that property does not live in any single file.

In short: Checklists find missing things. What survives is present, plausible, and wrong only in combination.

Identity is not ownership

Most teams check authentication. Fewer check that the row belongs to the caller.

GET /api/invoices/:id
  requireAuth(req)                 // who are you?
  return db.invoices.find(id)      // here, have an invoice

Auth is present, so a scanner sees a protected endpoint. It is also a plain IDOR: any logged-in user can walk :id and read every invoice you store. The fix is to make ownership part of the query rather than a check after it:

  user = requireAuth(req)
  return db.invoices.find({ id, orgId: user.orgId })

That is not a style preference. A post-fetch comparison depends on every future developer remembering to write it, and it leaks existence through response timing. Scoping the query means the wrong row never loads.

Four places this reliably hides:

• Nested routes. /projects/:pid/tasks/:tid where pid is authorized and tid is fetched by id alone. Pair your own valid pid with someone else's tid and the check passes.
• Bulk endpoints. The single-item route scopes correctly. The array variant added later checks the first id, or none.
• Exports. The list view filters by org. The CSV export runs a different query that does not.
• Second write paths. PATCH /comments/:id is scoped. The admin route or GraphQL resolver touching the same record is not.

The underlying mistake is enforcing authorization per endpoint instead of per resource, so every new route is a fresh chance to forget. Put it where queries cannot bypass it: a repository layer that demands a tenant scope, or row-level security in the database.

The three races you will actually hit

Check-then-act. The most common shape in generated code:

const existing = await db.users.findByEmail(email)
if (existing) return error('taken')
await db.users.create({ email })

Two simultaneous signups both read "nothing there," both insert. Re-checking does not help, it just moves the window. Only a unique constraint in the database makes this atomic; your job is to catch the violation and return a clean error.

Read-modify-write. Fetch a balance, compute in application code, write it back. Two requests each read 100 and each write 90, so you have lost 10. Do the arithmetic in the database (UPDATE ... SET balance = balance - 10 WHERE balance >= 10) and check rows affected.

Non-idempotent handlers. The user double-clicks. The mobile client retries on bad wifi. Your payment provider redelivers the webhook, which every provider does by design, because they guarantee at-least-once. Without an idempotency key, each delivery does the work again.

One question catches all three: what happens if this exact request arrives twice, at the same instant? If correctness depends on them arriving in order, you have a bug.

How one slow dependency takes everything down

The intuitive response makes this worse, which is why it is worth tracing.

A dependency slows from 50ms to 5s. Calls hold their connections longer. Your pool fills. Requests that never touch that dependency now block waiting for a slot. Your health check shares the pool, so it times out too. The orchestrator concludes the container is dead and restarts it. The replacement starts cold and sends more load to the dependency that was already struggling. Retries accelerate every step.

What breaks the spiral:

• A timeout on every network call, shorter than your caller's. Most HTTP clients default to unbounded, which is the wrong default for a server.
• Backoff with jitter, and only on idempotent operations. Fixed-interval retries synchronize clients into a herd that arrives together every cycle.
• A retry budget capped as a share of total traffic. Per-request retries with three attempts means a struggling dependency gets triple load exactly when it can least take it.
• Circuit breakers. Failing in 1ms instead of 30s is what keeps the pool free.
• Separate pools per dependency, so a slow one cannot starve the rest of the app.
• A health check that touches nothing shared, or the platform will restart healthy processes.

The design question underneath: for each dependency, what should users see when it is down? If the honest answer is "an error page," that dependency is load-bearing for your entire product.

The failures that never raise an error

Partial writes. Create order, decrement stock, charge card. If the charge fails and the first two were not in a transaction, you have an order for inventory you removed and were not paid for. Nothing errored, from the application's point of view.

Dual writes. Writing to your database and then to a search index or cache is a distributed transaction in disguise. When the second write fails the two disagree permanently and silently. Write once, derive the rest from an outbox or log.

The deploy window. Schema changes and the code using them do not land at the same instant. For a minute or two, old and new code both serve traffic. A migration that drops or renames a column breaks every old instance still running. Hence expand-and-contract: add the column, write both, backfill, switch reads, drop the old one in a later deploy.

Restores you have never run. A backup you have not restored is a file, not a backup. The failure modes are mundane: the job stopped silently in March, the dump excludes a schema, or the restore takes eleven hours you did not budget for.

Why review does not catch these

Reading a diff and finding it sensible tells you almost nothing here, because each piece genuinely is sensible. IDOR looks like a normal fetch. Check-then-act looks like careful validation. A missing timeout looks like an ordinary HTTP call. There is no anomaly to spot.

What surfaces them are questions about interaction, not inspection:

• What if this runs twice, simultaneously?
• What does this return when the id belongs to someone else?
• What if this dependency takes 30 seconds instead of 50ms?
• If this fails halfway, what state is left behind?
• What is running during the deploy window?

Ask these in your prompts too. A model will implement scoped queries, idempotency keys and timeouts correctly when asked directly. It will not volunteer them.

What a scanner can and cannot tell you

Worth being precise, because these are different jobs. Scanning the deployed site is authoritative on the presence class: which headers are actually set, what your live certificate chain looks like, which pages error or redirect oddly, whether mixed content loads, which routes are unexpectedly public. It sees deployed reality rather than what your config claims, and it keeps checking after you have moved on.

It cannot tell you an endpoint returned the wrong tenant's row, because that response is a valid 200 with plausible data. It cannot see a race condition, which needs concurrency to appear.

So they are complements. Automate the mechanical layer completely, then spend your own attention on the interaction failures that need a person to reason about them. Manually verifying header presence is the wrong use of the scarcer resource.

Final Thoughts

Finish the presence checklist. It matters, it is finite, and it is the part machines do better than you.

What is left needs someone holding the system in their head: how requests interleave, what happens when a dependency is slow rather than down, which invariants must hold across operations that know nothing about each other. None of that shows up in a diff, and none of it is detectable by scanning, because nothing is missing.

Automate every check that can be automated so coverage of the mechanical layer is free. Then ask the five questions above. They find more real bugs than any checklist.