Overload and recovery testing

Tests whether a service sheds excess work and returns to normal after load drops instead of remaining saturated.

overload-recovery-testing

See how it helps

When is it useful?

Use when traffic spikes cause prolonged failures even after users leave.

What does it help you do?

Checks bounded queues, concurrency and cheap rejection paths.
Examines timeout isolation and retry amplification.
Measures recovery time and remaining backlog after overload ends.

Try asking this

Run an authorized bounded overload test, reduce traffic and verify latency and queues return to baseline. Observe queue depth while excess requests are rejected, then return to normal traffic. Verify backlog drains without a retry storm or permanently elevated latency.

Detailed documentation

Read the skill’s original instructions, examples and checks in their authored language.

Details…Show less

Overload & Recovery Testing

When to Use

Use when verifying that a service survives traffic beyond its maximum capacity without crashing, corrupting data, or entering a permanent zombie/metastable failure state, and recovers promptly once load subsides.

Why Systems Fail to Recover After Overload (Metastable Failures)

When arrival rate $\lambda$ exceeds capacity $\mu$, naive systems suffer congestion collapse:

Queues and connection pools fill up; latency climbs above client timeouts.
Clients time out and retry immediately, multiplying arrival rate ($2\times$–$5\times$ retry amplification) even after real user traffic drops.
The server wastes 100% of CPU processing already-abandoned requests from the back of an unbounded queue, or crashes with OOM / exhausted file descriptors.

Five Overload & Recovery Mechanisms to Verify

1. Fast, Cheap Load Shedding & Rate Limiting

When concurrency or arrival rate exceeds safe capacity, the system must reject excess work immediately and cheaply (HTTP 429 Too Many Requests or 503 Service Unavailable with Retry-After) at the edge or admission controller—before allocating heavy memory, DB connections, or worker threads.
Verify under overload that:
- Shed requests fail fast (e.g., < 15ms).
- Admitted requests up to capacity continue to succeed within SLO (goodput preservation under overload).

2. Bounded Queues, Concurrency Limits & Lifo/Timeout Eviction

Every in-memory queue, thread pool, and connection pool must have an explicit upper bound (maxQueueSize, maxConcurrency).
Verify that requests waiting longer than their deadline or client disconnect (AbortSignal / closed socket) are dropped before execution rather than processed after the caller has already timed out.

3. Downstream Timeout, Bulkhead & Circuit Breaker Isolation

Simulate a slow or failing downstream dependency (e.g., third-party API latency raised from 50ms to 5s).
Verify:
- Strict downstream timeouts prevent a single slow integration from tying up all server workers (bulkhead isolation).
- Non-critical features degrade gracefully while core user flows remain responsive.

4. Retry Backoff, Jitter & Retry Budgets

Verify that internal SDKs, background workers, and frontend clients use exponential backoff with full jitter and respect a bounded retry budget (or stop retrying on 4xx / 429 without Retry-After).

5. Post-Overload Recovery Verification (Mandatory Phase)

Never end an overload test at peak traffic! Always include a Recovery Verification Window:

Ramp traffic above capacity ($1.5\times$–$2\times$ $\lambda_{\text{max}}$) for a bounded window.
Step traffic back down to normal baseline ($0.5\times$ $\lambda_{\text{max}}$).
Measure Recovery Time ($T_{\text{recovery}}$) and assert that:
- Error rate drops back to 0%.
- p95 and p99 latency return to pre-overload baseline.
- Memory, open connections, and queue depths drain back to normal steady state without requiring a manual process restart.

Included files 1

SKILL.md