Reliable workers and scheduled jobs

Workers & Scheduled Operations

Background workers and scheduled cron jobs execute critical business side effects out of sight of HTTP load balancers. If they are killed mid-job during deployments or run concurrently without locks, they silently corrupt state or duplicate external actions.

Graceful Shutdown & Drain Lifecycle

When deploying a new version or scaling down a worker node:

1. Stop pulling new work immediately on SIGTERM / SIGINT:
   • Close the queue consumer subscription so no new jobs are dequeued while the worker is shutting down.
2. Finish in-flight jobs within a bounded drain window:
   • Configure the process supervisor (stop_grace_period in Docker/Compose, terminationGracePeriodSeconds in Kubernetes, kill_timeout in Fly/systemd) to exceed the maximum normal job execution time (e.g., 30s–120s).
   • If a job cannot finish before the grace deadline, checkpoint its progress, release its lease cleanly so another worker can resume, and exit before SIGKILL.
3. Separate worker health checks from HTTP server checks:
   • A worker process with no HTTP server still needs a liveness/heartbeat signal (e.g., updating a local heartbeat timestamp file or /healthz socket only after the consumer loop tick succeeds and Redis/broker connection is healthy).

Idempotency, Concurrency & Mixed-Version Rollouts

• Design every job and cron task to be idempotent:
  • Queue brokers guarantee at-least-once delivery, and cron schedulers can fire twice during clock skew or leader election. Key external side effects (emails, charges, webhooks, DB writes) by a deterministic idempotency key or status transition guard (WHERE status = 'pending').
• Prevent overlapping cron executions:
  • Use a distributed lock with a bounded TTL (SET resource val NX EX ttl or Postgres advisory lock) or scheduler concurrencyPolicy: Forbid so a slow 6-minute report job scheduled every 5 minutes does not pile up 20 concurrent overlapping runs.
• Cross-version payload compatibility during deploys:
  • During a deployment, Version $N+1$ of the web app may enqueue jobs that are picked up by a still-running Version $N$ worker (or vice versa). Never rename or remove required job payload fields in a single step: add new optional fields first, deploy workers that understand both old and new shapes, and remove legacy fields in a later release.

Dead-Letter Queues (DLQ) & Backpressure

• Enforce bounded retry counts with exponential backoff + jitter (attempt 1..N), and route poison-pill messages that exhaust retries to an inspectable Dead-Letter Queue (DLQ) with the failure reason, stack trace, and payload metadata preserved for replay once fixed.