Retries and compensation

Retries & Compensation

Use this skill when designing error handling, backoff retry policies, idempotency guarantees, manual recovery, and Saga compensations in workflows.

Step-Scoped Retries & Idempotency (Helland & Hohpe)

1. Retry the Failing Step, Never the Whole Workflow from Scratch:
   • When step $S_k$ fails after steps $S_1 \dots S_{k-1}$ succeeded, checkpointed upstream results remain intact. Retry only $S_k$ according to its step-level policy (maxAttempts, initial delay, exponential backoff, jitter).
2. Deterministic Idempotency Keys for External Side Effects:
   • Never promise "exactly-once" external execution without an idempotency mechanism.
   • Derive a deterministic idempotency key (hash(tenantId, runId, stepExecutionId)) that stays identical across retry attempts of the same step execution, and pass it to downstream APIs (payments, emails, ticket creation, order placement).
3. Separate Retryable vs. Non-Retryable Failures:
   • Retryable: Transient network timeouts, 429 Too Many Requests, 502/503/504 gateway errors.
   • Non-Retryable (Fail Immediately): Validation errors (400/422), permission denials (401/403), missing resources (404), or business rule rejections.
4. Surface Uncertain External Outcomes Explicitly:
   • If a non-idempotent external call times out after the request was sent, do not blindly re-send a second charge or message. Mark the attempt as uncertain_side_effect or query the remote provider's status before retrying or escalating to operator review.

Saga Compensation & Partial Failure Recovery

• Compensating Steps Are Forward Business Actions, Not Magic Time Travel:
  • When a multi-step workflow fails at step 4 after steps 1–3 committed external changes (e.g., reserved stock, created invoice draft, provisioned user), execute explicit compensating steps in reverse topological order (ReleaseStock, VoidDraftInvoice, DeprovisionUser).
  • Record compensation status (compensating $\to$ compensated | compensation_failed) distinctly in the run history, and make compensating handlers idempotent and retryable themselves.
• Controlled Operator Replay (Retry from Failed Step):
  • Allow authorized operators to retry a failed step or resume a paused run from its last durable checkpoint with full audit logging.

Checklist

• Are retries scoped to the individual failing step using a stable per-step idempotencyKey rather than re-running completed prior steps?
• Are non-retryable 4xx/validation errors separated from transient 5xx/429 errors?
• Are post-dispatch timeouts on non-idempotent external calls flagged or reconciled before retrying?
• Are Saga compensations modeled as explicit, idempotent reverse steps with their own tracked execution status?