Queue and worker load testing

Queue & Worker Load Testing

When to Use

Use when load testing background job processors, event-driven pipelines, webhook consumers, media/document workers, or scheduled batch workflows (BullMQ/Redis, RabbitMQ, SQS, Kafka, Celery, Laravel Horizon/Queue, Sidekiq, Postgres/SQLite job tables).

Queueing Fundamentals & Core Metrics

HTTP response time (202 Accepted) only measures how fast a job was enqueued—it says nothing about whether workers can keep up! Always measure the complete asynchronous lifecycle:

1. The Four Essential Queue Metrics

1. Enqueue / Arrival Rate ($\lambda_{\text{in}}$): Jobs or messages added per second.
2. Completion / Drain Rate ($\mu_{\text{out}}$): Jobs successfully processed per second.
3. Queue Depth / Backlog ($L_q$): Number of waiting + active jobs. Whenever $\lambda_{\text{in}} > \mu_{\text{out}}$ for duration $T$, backlog grows at $(\lambda_{\text{in}} - \mu_{\text{out}}) \times T$.
4. Oldest Unprocessed Job Age / End-to-End Queue Latency ($W_{\text{total}} = t_{\text{completed}} - t_{\text{enqueued}}$): The true user-facing SLI. Even a small backlog causes huge latency if workers stall on slow jobs.

2. Backlog Build-Up & Drain-Time Verification

Test both steady-state sustainability ($\lambda_{\text{in}} \le \mu_{\text{out}}$) and burst drain recovery:

• Inject a realistic burst batch (e.g., $5,000$ jobs at $\lambda_{\text{in}} = 250\text{/s}$ while workers process at $\mu_{\text{out}} = 50\text{/s}$).
• Measure:
  • Peak queue depth and memory footprint in the broker (Redis used_memory, SQS/RabbitMQ ready messages, DB queue table lock contention).
  • Drain Time ($T_{\text{drain}}$): Time required to bring queue depth back to 0 once burst ingestion stops.
  • Head-of-line blocking: Do slow/heavy jobs starve fast, latency-sensitive jobs (e.g., password reset emails stuck behind bulk PDF exports) because they share a single queue or worker pool?

3. Concurrency, Lock Contention & Visibility Timeouts

• Tune worker concurrency against downstream bottlenecks: increasing worker threads from 10 to 100 often reduces throughput if all 100 workers contend for a 20-connection database pool or hit third-party API rate limits.
• Verify Lease / Visibility Timeout / Lock Renewal (lockDuration / stalledInterval):
  • When a job takes longer under high load (e.g., 45s vs normal 5s), check whether its lock expires prematurely, causing another worker to pick up the same job concurrently.
• Assert Idempotency & At-Least-Once Correctness:
  • Verify that duplicate deliveries or stalled-job retries do not produce duplicate side effects (double billing, duplicate notifications, corrupted state).

4. Poison Messages, Retry Storms & Dead-Letter Queues (DLQ)

• Inject controlled downstream failures (e.g., 10% upstream 503 or malformed payloads) during load:
  • Confirm retries use exponential backoff with jitter, not immediate tight-loop retries that multiply queue load (retry amplification).
  • Confirm permanently failing jobs transition to the Dead-Letter Queue (failed / DLQ) after maxAttempts without blocking healthy jobs.