Túlterhelés és helyreállítás tesztje

Overload & Recovery Testing

When to Use

Use when verifying that a service survives traffic beyond its maximum capacity without crashing, corrupting data, or entering a permanent zombie/metastable failure state, and recovers promptly once load subsides.

Why Systems Fail to Recover After Overload (Metastable Failures)

When arrival rate $\lambda$ exceeds capacity $\mu$, naive systems suffer congestion collapse:

1. Queues and connection pools fill up; latency climbs above client timeouts.
2. Clients time out and retry immediately, multiplying arrival rate ($2\times$–$5\times$ retry amplification) even after real user traffic drops.
3. The server wastes 100% of CPU processing already-abandoned requests from the back of an unbounded queue, or crashes with OOM / exhausted file descriptors.

Five Overload & Recovery Mechanisms to Verify

1. Fast, Cheap Load Shedding & Rate Limiting

• When concurrency or arrival rate exceeds safe capacity, the system must reject excess work immediately and cheaply (HTTP 429 Too Many Requests or 503 Service Unavailable with Retry-After) at the edge or admission controller—before allocating heavy memory, DB connections, or worker threads.
• Verify under overload that:
  • Shed requests fail fast (e.g., < 15ms).
  • Admitted requests up to capacity continue to succeed within SLO (goodput preservation under overload).

2. Bounded Queues, Concurrency Limits & Lifo/Timeout Eviction

• Every in-memory queue, thread pool, and connection pool must have an explicit upper bound (maxQueueSize, maxConcurrency).
• Verify that requests waiting longer than their deadline or client disconnect (AbortSignal / closed socket) are dropped before execution rather than processed after the caller has already timed out.

3. Downstream Timeout, Bulkhead & Circuit Breaker Isolation

• Simulate a slow or failing downstream dependency (e.g., third-party API latency raised from 50ms to 5s).
• Verify:
  • Strict downstream timeouts prevent a single slow integration from tying up all server workers (bulkhead isolation).
  • Non-critical features degrade gracefully while core user flows remain responsive.

4. Retry Backoff, Jitter & Retry Budgets

• Verify that internal SDKs, background workers, and frontend clients use exponential backoff with full jitter and respect a bounded retry budget (or stop retrying on 4xx / 429 without Retry-After).

5. Post-Overload Recovery Verification (Mandatory Phase)

Never end an overload test at peak traffic! Always include a Recovery Verification Window:

1. Ramp traffic above capacity ($1.5\times$–$2\times$ $\lambda_{\text{max}}$) for a bounded window.
2. Step traffic back down to normal baseline ($0.5\times$ $\lambda_{\text{max}}$).
3. Measure Recovery Time ($T_{\text{recovery}}$) and assert that:
   • Error rate drops back to 0%.
   • p95 and p99 latency return to pre-overload baseline.
   • Memory, open connections, and queue depths drain back to normal steady state without requiring a manual process restart.