systematic debugging

four phases, in order, each complete before the next:

1. Root-cause investigation — read errors completely, reproduce consistently, check recent changes, in multi-component systems instrument every boundary (log what enters and exits each layer) so one run shows where it breaks. trace bad values backward up the call stack to their origin.
2. Pattern analysis — find a working example of the same thing, read the reference implementation completely, list every difference from the broken one, however small.
3. Hypothesis and test — one specific, written hypothesis; smallest possible change to test it; one variable at a time. a failed hypothesis produces a new hypothesis, never a stacked fix.
4. Implementation — failing test reproducing the bug, single fix addressing the root cause, verify nothing else broke.

tripwire: three or more failed fixes means stop fixing and question the architecture. each fix revealing a new problem in a different place is the signature of a wrong structure, not a wrong line.

guess-and-check thrashes — slower than investigation, every time:

• each speculative fix is a change someone must unwind before the real cause is visible.
• retries and timeouts bury the original error.
• a fix that "works" removes the pressure to investigate while the fault stays.
• after three failed fixes in different places, the evidence is contaminated and the structure — not a line — is the suspect.

red flags: "quick fix for now, investigate later", "just try changing X" — the self-test against entering that spiral.

• proposing a fix before Phase 1 is complete is the violation, even if the fix turns out right.
• fixes come one at a time, each verified, no bundled improvements.
• "No root cause found" is 95% of the time an incomplete investigation.

boundary instrumentation is phase 1 in code — one run then shows where the bad value enters:

// instrument every boundary before changing anything
async function handleWorkflowStart(command: StartCommand) {
  log.info("workflow.start.received", { command });
  const config = await providerAdapter.resolve(command.providerId);
  log.info("workflow.start.adapter_output", {
    providerId: command.providerId,
    config,
  });
  const result = await orchestrator.start(config, command);
  log.info("workflow.start.orchestrator_output", { result });
  return result;
}

// one failing run proves which boundary first produced the bad value.
// the failing test reproducing the bug is written before the fix.

any bug, test failure, unexpected behavior, performance problem, build failure, or integration issue — especially under time pressure, when the process matters most.

bad

symptom: workflow start failed
fix: add a retry (two attempts) and redeploy
result: workflow still fails, now tries twice and fails twice
cost: wasted two invocations, harder to diagnose because error is buried in retry logic
next fix attempt: maybe the timeout is too short? add more delays
three retries later: workflow fails predictably on every attempt
actual root cause (finally found): required binding was never deployed

retrying a systematic failure (missing configuration) only delays discovery: each guess burns cost, adds complexity, and contaminates the evidence — every stacked retry, timeout, and workaround must be unwound before the true cause is visible.

good

symptom: workflow start failed
investigation phase 1: read error message, see configuration-missing
evidence: provider adapter returned configuration-missing diagnostic
phase 2: reproduce reliably and instrument the boundary
observed fact: the binding is never read by the worker
hypothesis: required binding is absent in deployment
phase 3: check recent changes and deployment logs
found: the binding was removed in the last migration but not re-created
fix: add binding to deployment
verify: workflow starts and completes successfully

investigation finds the cause before any change. the single targeted fix either works or shows a different, real error; nothing speculative to remove later; the same failure class stays fixed everywhere. debugging time shrinks with each root-cause fix instead of growing.

debuggability first, repair second