incident response

flowchart LR
    A[detect: symptom + correlation id] --> B[communicate: one status line]
    B --> C[stop the harm: flag · limit · rollback]
    C --> D[diagnose: logs by correlation id · hypothesis log]
    D --> E[fix: one change · live-boundary verify]
    E --> F[close: root cause + detection note]
    B --> B2[next update time]

why

every minute of an incident is compounding customer cost. invariants:

ongoing harm stops before diagnosis.
one honest status line exists throughout.
"resolved" is backed by a live probe — never a deploy log.
silence during an outage is itself a failure: customers assume the worst and work around you.
speculative fix attempts destroy the evidence needed for root cause.

the loop buys stopped harm and a truthful record at the cost of diagnostic patience.

rule

an incident is an unexpected product behavior that harms people now. work it with the same discipline as debuggability-first repair: correlation id first, explicit hypotheses, root cause before fix, evidence at every claim. the additional incident obligations are communication and stopping the harm:

silence teaches people the system is broken and the team is not in control — they stop using the product, work around the outage, or move data elsewhere. a 30-minute outage with no status update feels like 2 hours.
investigate first, stop harm second, and customers keep losing transactions or data while you debug — a preventable cost.

response loop

phase	obligation
detect	a person or an alert names the symptom with a correlation id, time, and scope.
communicate	one status line exists: what is broken, for whom, since when, next update time. update it on every material change — silence reads as failure.
stop the harm	if a feature switch, rate limit, or rollback path can stop ongoing harm, take it before root-causing.
diagnose	logs/traces by correlation id, per correlation tracing; hypothesis log with falsified hypotheses recorded.
fix	one change, verified at the live boundary before the status line says resolved, per verification without tests.
close	post-incident note: root cause, why monitoring didn't catch it, which guru file (if any) would have prevented it.

the status line

the status line is a truthful product state at the operations scale: never invents progress, never claims recovery before evidence, never hides scope. "investigating; next update 15m" is honest; "resolved" while the probe still fails is the incident's second failure.

do not

attempt multiple speculative fixes in sequence — one hypothesis, one change, verify, then the next.
mark resolved from a deploy log; probe the boundary first.
end an incident without a written note, including why detection missed it.
hide a customer-facing impact because the internal error rate looks small.

verification

run one tabletop: pick a plausible failure (provider down, bad deploy), walk the loop, confirm each phase has an owner and a concrete artifact (status line, hypothesis log, probe result, closing note). confirm the alert that should have fired exists — or the note says why monitoring missed it.

good and bad

bad

[10:12 UTC] workflow API returns 500 for some users
[10:24 UTC] we deployed a possible fix. should be resolved now.
[10:47 UTC] customer email: "I lost 2 hours of work, gave up and am using a competitor"
[11:15 UTC] team realizes the fix did not work; still investigating
[12:30 UTC] actual fix deployed; confirmed working
[13:00 UTC] incident resolved

nobody said what was broken, for whom, or when it would be fixed; after 12 minutes of silence customers assumed permanent breakage, worked around the outage, and did not come back.
a status line every 15 minutes (even "still investigating") would have kept people informed.
stopping harm at 10:12 (disable the broken route, serve stale cache, or roll back) would have prevented the customer's 2-hour loss and the 2-hour debug time.

good

investigating failed workflow starts since 10:12 UTC; impact is new runs only.
correlation: op_abc123. next update: 10:30 UTC.

communicates the known fact while investigation and the live repair probe continue.
customers plan around a bounded, honest update schedule instead of assuming permanent breakage.
recorded scope + correlation id give the repair a starting point, not a cold search.
resolution is claimed only after a live-boundary probe, so the closing record is trustworthy input to the post-incident note — root cause, detection gap, and the guru that would have prevented it.