Durable waits and events

Durable Waits & Event Correlation

Use this skill when implementing workflow delays, scheduled wakeups, webhook/event listeners, timeout races, and crash-safe resumption.

Never Hold Long Waits in Volatile Process Memory (Kleppmann)

1. Persist Wait State Before Releasing the Worker:
   • Never implement a workflow delay or external event wait with setTimeout or an in-memory promise that vanishes on process restart or deploy.
   • Persist a durable wait record (runId, stepExecutionId, waitType, wakeAt, correlationKey, timeoutAt, state: 'waiting'), commit the step checkpoint, and release the worker thread.
2. Idempotent Timer & Event Wakeup:
   • When the scheduler or event router fires at t >= wakeAt (or an incoming event matches correlationKey), claim the transition atomically (UPDATE ... WHERE state = 'waiting') so concurrent workers or duplicate scheduler ticks wake the step at most once.
   • On wakeup, resume execution from the persisted step checkpoint without re-executing already completed upstream steps.

Event Correlation, Buffering & Deduplication

• Explicit Correlation Keys & Tenant Scoping:
  • Match incoming events/webhooks to waiting steps using a composite scope: (tenantId, eventType, correlationKey) (e.g., orderId, paymentIntentId, ticketId). Never allow an event from tenant $A$ to wake a waiting run in tenant $B$.
• Deduplicate Repeated Events:
  • Record processed eventId / delivery keys so webhook retries do not trigger duplicate resumptions.
• Early / Out-of-Order Event Arrival Race Protection:
  • An external webhook (e.g., payment confirmation) can arrive milliseconds before the workflow step finishes persisting its waiting subscription.
  • Buffer recent correlated events in an inbox table or check the event log transactionally when registering the wait so early arrivals are matched immediately instead of hanging until timeout.
• Bounded Event Waits:
  • Every WaitForEvent step must support an explicit timeoutAt deadline routing to a timeout port or failure handler so runs cannot remain orphaned forever.

Checklist

• Are all workflow delays and event waits persisted in durable storage rather than held in worker memory?
• Is the transition from waiting to running claimed atomically so duplicate ticks or webhooks cannot double-resume a run?
• Are correlation keys strictly scoped by tenantId and protected against early/out-of-order webhook arrivals?
• Does every external event wait define an explicit timeout deadline and timeout outcome path?