retries and idempotency

why

retry recovers from an uncertain attempt — the effect may have succeeded before the failure was observed. without a stable idempotency key derived from a durable business fact, each retry looks like a brand-new request to the provider: duplicate charges, duplicate messages, duplicate artifacts. the key makes "already done" provable.

rule

every retryable side effect has a stable idempotency key owned by a durable business fact or provider contract. retry is recovery from an uncertain attempt, never permission to duplicate money, messages, artifacts, or state transitions.

classify the result

retryable failure: transient provider, network, rate, or infrastructure failure. preserve the cause, throw to the workflow's retry policy.
correctable failure: invalid input, expired approval, missing authority, unmet prerequisite. persist a typed blocked/rejected outcome with a remediation path; do not retry blindly.
permanent failure: invariant or integrity failure. persist diagnostic context, end the run visibly.
completed result: persist once under its stable identity, return the reference for later steps.

keys

use a business identity the operation naturally owns: runId + stage, an external provider request identity, or a unique event identity. enforce it in the durable store or provider request, not a process-local map.

never derive the key by rewriting display strings — build it from typed canonical identifiers or a named boundary builder.

attempt lifecycle

moment	durable action
before side effect	read/claim the business identity and record the attempt context.
provider accepts idempotency key	persist the provider reference or accepted fact before reporting progress.
uncertain timeout	retain the same business identity; reconcile or retry through the provider's idempotent boundary.
classified retryable failure	preserve cause and let owned retry policy schedule the next attempt.
correctable/permanent failure	persist terminal/blocked state with remediation; do not schedule unchanged work.
repeat delivery after completion	return/read the existing durable outcome rather than perform the side effect again.

good and bad

bad

try {
  await provider.charge({ amount });
  // persist success
} catch (error) {
  log(error); // logged but no durable identity
}
// report the step as completed, attempt #1

the provider charges the customer, then times out before returning. the code catches the timeout, logs it, marks the step completed. the workflow retries: it charges again (the first charge is unknown to it). two charges, no way to reconcile — no durable identity ties either attempt to the business fact, so recovery is a manual refund decision instead of automatic deduplication.

good

const key = `${runId}:charge`;
const result = await provider.charge({ amount, idempotencyKey: key });
await persistAcceptedFact(runId, key, result.reference);
// on retry, the provider's own idempotency rejects or returns the same result

the side effect carries a stable key derived from a durable business fact (run + stage). on retry after a timeout, the provider sees the same key and returns the same result or rejects as duplicate. one charge recorded durably, one external effect.

verification

force duplicate delivery before and after the external side effect, then force a timeout where provider completion is uncertain. confirm both paths retain one business fact and one external effect, with attempt history/diagnostics that explain whether reconciliation, retry, or correction is required.

do not

catch an exception, log it, and claim the step completed.
retry authorization, validation, or integrity failures without a changed input or state.
make the attempt number the only identity of a business side effect.
rely on the workflow engine alone to make a third-party API idempotent.