Rollback before you deploy: a Friday checklist

Deploy is not the hard part

Most teams can copy an artifact to a server. The failure we see is a migration that cannot run backward, a config change that is not in git, or a cache that still serves the new code’s JSON after the binary is reverted. We treat rollback as a rehearsed path, not a wiki page written after the outage.

What we require before a production push

• Artifact identity. The previous version is a tagged image or a named build, not “whatever was on the box yesterday.”
• Forward-compatible schema. Add columns as nullable (or with defaults) in a release that does not yet require them. Expand, deploy app, then constrain. Never expand-and-break in one step.
• A flag for the risky behavior. New query path, new payment provider, new report — off by default. Rollback can be a flag flip while we keep the new binary if the binary itself is fine.
• Health checks that match user pain. HTTP 200 on /health is not enough if the dashboard query is timing out. We check the same endpoint the morning peak uses.
• One person who can abort without a meeting. If rollback needs three approvals, it will not happen at 6:20 AM.

We actually run rollback in staging

Once per release train we deploy to staging, then execute the abort steps: previous image, previous config, confirm the health URL, confirm a write still lands. If the down-migration is untested, we do not ship the up-migration. We have delayed Friday releases for this. Clients prefer a Monday ship over a weekend restore.

Shadow traffic, which we used on the dispatch cutover, is the same idea: new code sees copies of requests, old code still serves users, comparison happens before the flag flips. Details in the performance post and the CI/CD checklist.

What we put in the runbook

Four lines: how to flip the flag, how to redeploy the last image, who to page, and what metric means “done.” If the runbook is a 12-page PDF, nobody will use it during an incident. We store it next to the pipeline definition so it ships with the code.

This is the core of our CI/CD work. If your current “rollback” is restore-from-backup, we can put a real abort path in place without replacing your whole toolchain.