Security Reviews That Developers Don't Dread

There is no shortage of articles about Burp Suite. Most of them cover the happy path. This one is about what happened after the happy path ended.

The first symptom was subtle. Nothing failed outright, but secret rotation kept drifting in the wrong direction, and every fix seemed to make the next one harder.

Some background

When we finally measured, the numbers were worse than anyone guessed. Around 37% of the time spent on a typical change went into working around secret rotation rather than building anything.

The setup was ordinary: Burp Suite at the edges, Vault doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

Trying something different

We started by writing down the one rule we actually cared about, then deleted every bit of code that didn't serve it. It's not glamorous, but it works.

Stripped of the project-specific bits, it looks like this:

cosign sign --yes ghcr.io/acme/api@sha256:3f1c...
cosign verify \
  --certificate-identity-regexp 'https://github.com/acme/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/acme/api@sha256:3f1c...

Nothing about this is clever, and that's the point. Anyone on the team can read it in a minute and know exactly what it does.

A few things that helped along the way:

Agree on what "done" means before the first pull request.
Write the migration guide before writing the migration.
Measure before and after, even if it's just a rough number in a spreadsheet.

Trade-offs

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use OpenSSL for everything. That's a fair question with a boring answer: we tried, and it made secret rotation harder to reason about.

We still haven't solved supply chain security. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

Closing thoughts

That's it for now. Next time I'll dig into dependency pinning, which turned out to be closely related.