Home Projects Writing Contact About Apps

Writing

Mostly post-mortems on my own mistakes. Running the whole estate means being the person who gets paged, and the failures have taught me more than the features did.

Locked out of my own auth service

Five mistyped passwords, a permanent IP ban, and no way back except root SSH. Twice in ten minutes.

Every individual control was defensible — generic errors to prevent username enumeration, rate limiting, no secrets in logs. Stacked, they produced a failure that was total, permanent and silent, with a recovery path that assumed I still had root on the box.

  • Why the error was undiagnosable from both ends at once
  • How the mail server quietly became load-bearing
  • Recovery that does not depend on the thing that is down
Read it

The guard I wrote to prevent an outage

A startup check meant to stop a half-configured migration from taking everything down. It caused the outage instead.

Moving session signing from a shared secret to asymmetric keys, on a live estate, without logging anyone out. The check was right about the danger and wrong about the remedy — and the information it demanded was already inside the key it was refusing to start without.

  • Why a symmetric signing key is a privilege problem
  • When refusing to start is worse than the state it prevents
  • Misdiagnosing the next outage as my own change
Read it

Fork me on GitHub! 

Fork mutable on GitHub! 

Fork Luci on GitHub!