What a Leaked API Key Taught Our Team

Every codebase has one area everyone is a little afraid to touch. On our team, it was everything related to supply chain security.

Most of the complexity was accidental. Over the years we'd layered three different approaches to supply chain security on top of each other, and each one made sense at the time.

How we got here

The setup was ordinary: Burp Suite at the edges, Trivy doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

When we finally measured, the numbers were worse than anyone guessed. Around 23% of the time spent on a typical change went into working around supply chain security rather than building anything.

A simpler approach

Rather than a big-bang rewrite, we migrated one module at a time over about 4 weeks, keeping both paths alive behind a flag.

The core of it looks like this:

Content-Security-Policy: default-src 'self';
  script-src 'self' 'nonce-r4nd0m';
  style-src 'self';
  img-src 'self' data:;
  frame-ancestors 'none';
  base-uri 'self'

Nothing about this is clever, and that's the point. Anyone on the team can read it in a minute and know exactly what it does.

A few things that helped along the way:

• Delete dead code aggressively; it's the cheapest refactor there is.
• Keep the old path around until the new one has survived a real incident.
• Measure before and after, even if it's just a rough number in a spreadsheet.

Trade-offs

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Trivy for everything. That's a fair question with a boring answer: we tried, and it made supply chain security harder to reason about.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

Wrapping up

That's it for now. Next time I'll dig into secret rotation, which turned out to be closely related.