Reliable background jobs

Background Jobs

Ask before every state change: what happens if this runs again with the same input?

Duplicates are normal

• People double-click, refresh and resubmit.
• A call succeeds but its response is lost (timeout, reset connection), and the caller retries.
• Queues deliver at least once, webhook providers resend, scheduled runs overlap, workers crash after the work but before the acknowledgement. You cannot prevent duplicates; make every job and handler safe to run twice, concurrently, late or out of order.

Make repeats harmless

• Prefer operations that converge: set a state instead of applying a change. UPDATE status = 'paid' WHERE id = ?, an upsert or a delete by id are safe to repeat; balance = balance + x, an INSERT of a new row, sending an email or charging a card are not.
• Give each unit of work a stable key from its intent (the message or event id, the order id plus the action).
• Record the key under a unique constraint in the same transaction as the work. On a unique violation, the work was done: acknowledge and stop. A check followed by a write is a race; read the record from the primary database, not a replica.
• Acknowledge only after the transaction commits. On any other error, do not acknowledge, so the message is delivered again.
• External side effects get their own key derived from the job key, passed to the provider (charge-{key}). Without provider support, record the send and check the record first; know that a crash between sending and recording can repeat it once.

Small payloads, fresh state

• Pass ids and minimal data. Load the current state when the job runs and check it still applies (the order may be cancelled).
• Version the payload when its format changes; old messages may wait in the queue during a deploy. No secrets or personal data in payloads.
• Enqueue after the database transaction commits, or through an outbox: a job must never point at data that rolled back.

Retries and failure

• Transient errors (timeout, 5xx, 429, lock) retry with exponential backoff and jitter, up to a limit. Permanent errors (validation, not found, other 4xx) fail at once.
• After the last attempt, move the job to a dead letter queue or a failed state with its error, make it visible (alert, admin view), and provide a safe way to replay it. A failure is never silent.
• Rows marked processing by a worker that died must not stay stuck: give them a lease with an expiry, after which another worker may retry them, or mark them failed.

Partial failure and long work

• Multi-step jobs record progress so a retry resumes instead of repeating finished steps, or split into chained jobs that are each idempotent.
• Batches run in chunks with checkpoints; one bad item is recorded and skipped, never blocking the rest.

Timing and concurrency

• A job's timeout is shorter than the queue's visibility timeout or lock, or the job extends its lease while working.
• Scheduled tasks that must not overlap take a lock with an expiry, or run as a unique job.
• Workers shut down gracefully: on SIGTERM they finish or release the current job.
• Do not assume global ordering; where order matters, sequence per key or check versions.

Observe and test

• Log start, end, duration, attempt and outcome with the job id and key; track queue depth, the oldest job's age and the failure rate.
• Test the same job twice, two copies at once, a failure midway followed by a retry, and a permanent error.

Before you finish

• Every job is idempotent with a stable key, claimed in the same transaction as the work.
• Acknowledgement only after commit; external calls carry idempotency keys.
• Only transient errors retry; final failures are visible and replayable.
• Stuck processing rows expire; overlapping runs are locked out.
• Duplicate, concurrent and partial-failure cases tested.