Spike, stress and endurance tests

Separates sudden bursts, capacity breakpoints and time-dependent degradation into distinct measurable load experiments.

spike-stress-soak-testing

See how it helps

When is it useful?

Use when average-load tests cannot explain outages or gradual slowdown.

What does it help you do?

Tests abrupt traffic changes and the system's recovery.
Raises bounded load to identify the first limiting resource.
Runs sustained workloads to detect leaks and accumulating queues.

Try asking this

Plan authorized spike and endurance tests, comparing immediate recovery with memory growth over sustained traffic. Set duration and stop limits before generating traffic. Compare resource levels after cooldown with the baseline to distinguish temporary saturation from a persistent leak.

Detailed documentation

Read the skill’s original instructions, examples and checks in their authored language.

Details…Show less

Spike, Stress & Soak Testing

When to Use

Use when testing system behavior beyond steady-state normal load: sudden traffic surges (spike), finding the breaking point (stress), or detecting slow resource leaks over extended durations (soak). Never run all three indiscriminately on every code change—select the profile that matches the specific risk.

1. Spike Testing (Flash-Crowd & Burst Resilience)

Objective

Verify how the system absorbs an instantaneous $3\times$–$10\times$ traffic jump (e.g., marketing blast, ticket drop, webhook storm, cron thundering herd) and how quickly it stabilizes afterward.

Execution & Evaluation

Profile: Steady baseline ($1\times$) $\rightarrow$ rapid ramp (5–15 seconds) to spike ($5\times$) $\rightarrow$ hold for 1–3 minutes $\rightarrow$ rapid drop back to baseline ($1\times$) $\rightarrow$ observe recovery window.
What to Measure:
- Connection pool acquisition timeouts and cold-start latency during the upward edge.
- Whether excess traffic is cleanly shed (429 Too Many Requests / 503 with Retry-After) or causes all in-flight baseline requests to time out together.
- Recovery Time ($T_{\text{recovery}}$): How many seconds after traffic returns to $1\times$ it takes for p95/p99 latency and error rates to return to pre-spike baseline.

2. Stress & Breakpoint Testing (Locating the Failure Mode)

Objective

Determine the exact load level where SLOs fail, which component breaks first, and whether the failure mode is graceful or catastrophic.

Execution & Evaluation

Profile: Stepwise arrival-rate increments (e.g., $+20%$ RPS every 2 minutes) with hard safety abort guardrails (abortOnFail if error rate $> 40%$ or target host crashes) — never ramp blindly to infinity.
What to Measure:
- Exact RPS/concurrency of the knee point (latency inflection) and SLO breach point.
- Whether throughput plateaus cleanly at capacity or collapses (congestion collapse / retrograde scaling) due to lock contention, GC thrashing, or retry storms.
- Data integrity under stress: zero partial writes, double-charges, corrupted state, or lost jobs.

3. Soak / Endurance Testing (Time-Dependent Degradation)

Objective

Detect slow leaks and cumulative degradation that short 3-minute load tests never reveal.

Execution & Evaluation

Profile: Sustained moderate-to-high load (60–75% of max capacity) over an extended window (or accelerated iteration count in CI/staging).
What to Measure:
- Memory Growth Slope: Plot heap used, RSS, and native/off-heap memory (Buffer / C++ addons) over time. After forcing or waiting for GC cycles, does memory plateau or climb monotonically (MB/hour leak rate)?
- Connection, File Descriptor & Ephemeral Port Leaks: Check open file descriptors (lsof / /proc/<pid>/fd), database connection count (pg_stat_activity), Redis clients, and unclosed HTTP sockets/streams.
- Storage & Table Bloat: Unbounded log files, uncleaned temp files (/tmp), growing WAL/dead tuples (PostgreSQL autovacuum falling behind sustained write load), or growing in-memory caches without TTL/LRU eviction.

Included files 1

SKILL.md