backups and recovery

• a backup is not data safety; only a restore is.
• invariant: every durable store has a tested path back — proven against today's real snapshot format, credentials, schema.
• without restore proofs, recovery ability silently decays with every unrelated change; the first test runs during the incident, while data is still being lost.
• a repeated restore drill converts hidden liability into a verified recovery claim.

• every durable store names its recovery story in the slice: what is backed up, cadence, where, tested path back.
• a backup never restored is a liability listed as an asset.
• at data loss/corruption the restore command fails: procedure written for a different storage system, schema changed, credentials rotated, snapshot corrupted.
• the team finds the restore broken while the service is offline and data is still being lost; hours debugging while customers wait.
• recovery eventually succeeds but exceeds the declared RPO — unknowable until disaster hit.

• derived stores must be rebuildable from durable facts by design; a projection that cannot be recomputed is a hidden primary store violating database design.

• declare per durable store: recovery point objective (recent data losable) and recovery time objective (restore duration).
• personal products may be loose — but declared: "we assumed the provider's default" is a decision made by nobody.

• treat provider snapshots as a recovery plan without a restore proof.
• back up derived stores instead of making them rebuildable.
• keep the only copy of a secret in a backup file.
• let a restore procedure exist only in one person's head or one chat thread; it belongs in the slice or this file's owning store.

• the restore proof is the whole standard: once per cadence, restore one snapshot into a fresh non-production instance.
• record: the command, point-in-time achieved, row counts of key tables, one application boundary exercised against the restored store.
• an unverified backup fails this section by definition.

bad

database snapshots: enabled on provider auto-snapshot
restore drill: never run
declared RPO: 24h
incident: 2026-08-24 10:15 UTC, database corruption detected
restore attempt: `restore_from_snapshot('daily_2026-08-24')`
result: restore command fails
reason: provider migrated snapshot format in the last schema update;
  the restore command expects the old format
resolution: manual reconstruction from binlog exports, 14 hours
actual RPO achieved: 18 hours
data lost: 3,600 customer workflows that started in hours 8–24
severity: customers lost work they expected to be saved
root cause: restore procedure was never exercised

• declared an RPO they could not defend; restore failed because never tested.
• manual recovery exceeded stated RPO by 6 hours; customers lost data.
• no way to know the backup was broken until the crisis.

good

weekly restore: fresh non-production database
proof: row counts + workflow status route
achieved rpo: 24h

• recovery claim backed by repeated restore + application-boundary check.
• cadence drill surfaces format drift, rotated credentials, schema changes as a failed weekly exercise, not a live incident.
• row counts + route probe prove the restored store is usable by the application — not merely present.
• declared RPO becomes a defensible, evidenced number.

schema migrations · database design · incident response · secrets and keys · verification without tests