The guard I wrote to prevent an outage

The migration

Sessions were signed HS256 with a shared secret. That is fine until you count who holds the secret: six services, five of which only ever need to check a token. With a symmetric algorithm the key that verifies also mints, so compromising the least important service on the estate would let you forge an admin session for all of it.

RS256 splits those capabilities. Auth holds the private key and is the only thing that can issue a session; everything else holds the public key and can only verify. Straightforward, except you cannot switch a live estate atomically — a seven-day session issued yesterday has to keep working. So verification accepts every key it holds, both algorithms at once, and the old secret retires on its own after a week.

The thing I was right to worry about

Auth does not only sign tokens. It verifies them too — six separate places: the cookie reader, the verify endpoint, the refresh endpoint, the enrolment and challenge routes, the session endpoint.

If signing moved to RS256 while verification stayed on the old secret, auth would mint tokens and then reject its own. Every session invalid at once. The login page would issue a token that failed on the next request. Every app on the estate goes down together, and the cause is one variable on one stack.

So I wrote a startup check: if the private key is present and the public key is not, refuse to start.

Why that was the wrong remedy

It converts a half-finished migration into an outage.

Worse, it does so at exactly the moment someone is midway through the rollout I had written — the instructions say public key everywhere first, private key second, and the failure fires precisely when you have done step two and are about to do step one on the remaining stacks. The guard triggers during the normal, expected, transient state of following the procedure.

And auth going down is not one service going down. It is the login provider for everything, so it is all of them.

The information was already there

An RSA private key contains its public key. It is derivable in one call:

crypto.createPublicKey(privateKey)
    .export({ type: 'spki', format: 'pem' })

Auth never needed to be told. The state the guard was protecting against — signing something it cannot verify — cannot arise, because it can always work out the other half of its own keypair. The derived key goes first in the list, ahead of anything configured, since it is the only one guaranteed to be the matching half.

The warning stayed, but it now says the true thing. The service at risk was never auth. It is the other services, which verify locally and cannot derive anything — until the public key reaches them, they reject every session the moment auth switches. That is worth a loud message. It is not worth refusing to boot.

A second mistake, ten minutes later

Auth then returned 502, and I was confident I knew why: I had just changed the startup path, so the startup path was the cause.

It was Cloudflare.

The recent change is the first suspect and usually the right one, which is exactly what makes it worth checking rather than assuming. I had a plausible mechanism and a matching symptom, and that felt like enough. The actual evidence — container logs — would have taken thirty seconds and pointed somewhere else entirely.

What I took from it

A guard that refuses to run is only correct if not running is safer than the state it prevents. I never checked that comparison. Signing unverifiable tokens is bad; auth being down is also bad, and it is bad immediately and completely. I had traded a hypothetical for a certainty.

Before you require configuration, check whether you can derive it. A required setting is a thing that can be forgotten, typo'd, or set on the wrong stack. The best fix for a footgun is usually to remove the trigger rather than to detect it.

Design for the transient state. I wrote a careful ordered rollout and then added a check that fires in the middle of it. If a procedure has intermediate states, those are real states the system will occupy, and they deserve as much thought as the endpoints.

A plausible cause is not a diagnosis. Especially when it is your own most recent change.

The whole flow is now covered by tests running under the exact configuration that broke it: private key present, public key absent — sign in, verify, cookie, account page, refresh, revoke. Deriving the key weakens nothing; a token signed by a different private key is still rejected, and week-old sessions on the old algorithm still verify.