durable orchestration

• a restarted run must make the same next decision the crashed run would have made.
• that holds only when every advance is derived from persisted typed facts — not memory, a previous model response, or live UI state; all evaporate exactly when durability is needed.
• a typed orchestrator over durable facts keeps business rules and provider mechanics replaceable underneath it.

• the workflow owns the durable process, not implementation details.
• it accepts a small typed start contract, advances named steps from durable facts, produces a typed outcome.
• a resumed run decides from persisted state — never memory, a previous model response, or a live ui.

• stage ordering, dependencies, retries, waits, cancellation, exhaustion.
• workflow instance identity, correlation identity, durable outcome.
• progress transitions as stable domain facts.
• starting bounded child workflows when one parent would own too much concurrent work.

• domain rules → a domain service, called from the step.
• provider calls → behind an adapter.
• user-facing rendering and live connection management → the Agent or ui.
• canonical records, artifacts, audit facts → their owning persistent store; a workflow engine is not the general-purpose source of truth.

type RunWorkflowInput = {
  runId: string;
};

type RunWorkflowOutcome = {
  runId: string;
  status: "completed" | "failed";
};

• pass stable references such as runId — not whole mutable records or prompt transcripts.
• inside the step: read the canonical record, validate its state, persist the next fact, return a bounded result.

• key question at every boundary: if this process restarts at this exact moment, which durable facts prove the only valid next decision?
• if the answer depends on memory, a UI payload, or an unrecorded response, the orchestration is not durable yet.

bad

step("decide next stage", async ({ runId, modelResponse }) => {
  if (modelResponse.choices[0] === "review") return startReview(runId);
});

• next stage transition depends on an untyped in-memory response → a resume cannot reconstruct the same decision.
• after a crash: restart from stale assumptions or stall entirely; a repeated provider response can advance the stage twice; nobody can audit why review over publish — the deciding input was never recorded.
• durable in name only.

good

step("decide next stage", async ({ runId }) => {
  const run = await loadRun(runId);
  const decision = validateTransition(adapter.parse(rawResponse));
  await persistStageFact(runId, decision);
  return decision.nextStage;
});

• reloads the canonical record, validates the response at a typed boundary, persists the fact that determines the next stage → a resumed run makes the same decision from durable state.

• verify more than the happy path.
• start with an existing run, interrupt it after each durable write, confirm a resumed execution reads the same canonical state and chooses the same next action.
• repeat a step request; confirm the idempotency boundary prevents a duplicate fact.
• force a classified dependency failure; confirm the outcome is durable, typed, visible to the owner.

• treat a workflow as a long function coordinating database tables, providers, prompts, and UI broadcasts inline.
• keep the only copy of a workflow decision in a closure or in Agent state.
• let a model response decide a state transition without typed validation and a durable domain write.
• continue after an unclassified provider failure as though the stage finished.

step boundaries and names · durable handoffs · async local execution context · durable event handoff