Reliable workers and scheduled jobs

Keeps background work recoverable during deployments and prevents overlapping schedules or repeated delivery from duplicating business actions.

workers-scheduled-operations

Agents using this skill

AgentOtto

See how it helps

When is it useful?

Use when jobs are interrupted by releases, scheduled reports overlap or failed messages need controlled recovery.

What does it help you do?

Stops accepting new work during shutdown and gives active jobs a bounded completion or recovery path.
Protects repeated actions and overlapping schedules while preserving compatibility between deployed versions.
Uses bounded retries and inspectable failed-job queues with useful worker health signals.

Try asking this

Prepare our report worker for deployment. An active report should finish or resume safely, the next scheduled run must not overlap and repeated delivery must not duplicate the result. Send shutdown during report generation and start the next scheduled attempt. Verify one final artifact, a bounded handoff and no overlapping execution for the same period.

Detailed documentation

Read the skill’s original instructions, examples and checks in their authored language.

Details…Show less

Workers & Scheduled Operations

Background workers and scheduled cron jobs execute critical business side effects out of sight of HTTP load balancers. If they are killed mid-job during deployments or run concurrently without locks, they silently corrupt state or duplicate external actions.

Graceful Shutdown & Drain Lifecycle

When deploying a new version or scaling down a worker node:

Stop pulling new work immediately on SIGTERM / SIGINT:
- Close the queue consumer subscription so no new jobs are dequeued while the worker is shutting down.
Finish in-flight jobs within a bounded drain window:
- Configure the process supervisor (stop_grace_period in Docker/Compose, terminationGracePeriodSeconds in Kubernetes, kill_timeout in Fly/systemd) to exceed the maximum normal job execution time (e.g., 30s–120s).
- If a job cannot finish before the grace deadline, checkpoint its progress, release its lease cleanly so another worker can resume, and exit before SIGKILL.
Separate worker health checks from HTTP server checks:
- A worker process with no HTTP server still needs a liveness/heartbeat signal (e.g., updating a local heartbeat timestamp file or /healthz socket only after the consumer loop tick succeeds and Redis/broker connection is healthy).

Idempotency, Concurrency & Mixed-Version Rollouts

Design every job and cron task to be idempotent:
- Queue brokers guarantee at-least-once delivery, and cron schedulers can fire twice during clock skew or leader election. Key external side effects (emails, charges, webhooks, DB writes) by a deterministic idempotency key or status transition guard (WHERE status = 'pending').
Prevent overlapping cron executions:
- Use a distributed lock with a bounded TTL (SET resource val NX EX ttl or Postgres advisory lock) or scheduler concurrencyPolicy: Forbid so a slow 6-minute report job scheduled every 5 minutes does not pile up 20 concurrent overlapping runs.
Cross-version payload compatibility during deploys:
- During a deployment, Version $N+1$ of the web app may enqueue jobs that are picked up by a still-running Version $N$ worker (or vice versa). Never rename or remove required job payload fields in a single step: add new optional fields first, deploy workers that understand both old and new shapes, and remove legacy fields in a later release.

Dead-Letter Queues (DLQ) & Backpressure

Enforce bounded retry counts with exponential backoff + jitter (attempt 1..N), and route poison-pill messages that exhaust retries to an inspectable Dead-Letter Queue (DLQ) with the failure reason, stack trace, and payload metadata preserved for replay once fixed.

Included files 1

SKILL.md