metrics and alerts

• instrumentation competes for the same scarce attention as code; alerts compete for human trust.
• a metric earns its place only by answering a named health question.
• an alert earns a page only when a person must act now — everything else trains responders to ignore pages until the alert that mattered is missed.
• ownership by question and boundary separates signal from a dashboard nobody reads.

• each metric answers a named product-health question, uses domain vocabulary, and is instrumented at its owning boundary.
• an alert fires only when a person must act now — it names the question, the boundary, and the first diagnostic step.

without owned questions:

• you instrument everything "might need it someday" — 300 metrics, all dashboard numbers, none actionable; at 3 a.m. you have no signal.
• alerting on single occurrences or averages of skewed distributions wakes the responder weekly for one-off noise; the alerts that matter drown in ignored pages.
• at incident time you cannot correlate which symptom is user-facing (error rate) vs internal (one retry spike) vs irrelevant (cold-start blip).
• two problems: nothing actionable gets built, and attention fragments across noise and signal until both get ignored.

instrument where the fact lives — a workflow backlog metric belongs to the workflow boundary that can count its own durable queue, not to a route that guesses.

• if no one would get woken up and act differently because of the alert, it is a dashboard number, not an alert.
• review alert noise on a cadence; an alert that fires weekly and is ignored daily is deleted, not muted.

• instrument everything instrumentable — each metric must name its question or be removed.
• alert on single occurrences or on averages of skewed distributions.
• create a metric with ad-hoc field names outside the domain vocabulary.
• leave an alert unmaintained once its question is stale.

• name the product-health questions in one place.
• for each question confirm: a metric exists, is owned by the right boundary, and its current value can be read from the live system.
• fire one deliberate threshold in a non-production environment; confirm the alert arrives with question, boundary, and first diagnostic step.

bad

alert: "error count > 0" — fires on every single occurrence, no action named
metric: avg(request_duration_ms) across all operations

• nobody can act on it; the average hides the tail — it pages weekly and is ignored daily.
• constant single-occurrence fires teach the responder to dismiss the channel; the alert that eventually matters arrives into a habit of silence.
• the averaged latency metric shows a healthy number while every p99 request times out — the real user-facing symptom is invisible in the one dashboard built to catch it.

good

metric: payment_route.error_rate by diagnostic code (owned by route edge)
alert: payment_route.error_rate > 2% for 10m
action: check provider status page, replay one probe charge

• each metric answers a named question at its owning boundary; the alert fires on rate, names the symptom, and tells the responder the first step.
• pages stay rare enough to be trusted, so attention concentrates on real incidents.
• the named question set gives the team a shared, bounded view of product health that survives staff and architecture changes.

error tracking · structured logging · incident response · fanout and capacity · verification evidence