Analyze performance measurements

Performance Measurement & Analysis

When to Use

Use when evaluating load test results, diagnosing latency degradation, correlating client metrics with server telemetry, or reviewing performance claims.

Core Analytical Discipline

1. Never Evaluate Latency by Averages Alone

• Arithmetic mean (avg) masks bimodal distributions and tail stalls. Always report p50 (median), p90, p95, p99, and max, alongside sample count (N) and error/timeout rate.
• Distinguish Service Time (actual work time inside the handler) from Wait / Queueing Time (time spent waiting for a connection pool slot, worker thread, lock, or event loop turn). When a resource approaches 100% utilization ($U \to 1$), queueing delay grows nonlinearly ($\propto \frac{U}{1 - U}$).

2. Separate Configured Load, Achieved Load, and Goodput

Always report three distinct rate metrics side by side:

1. Target / Configured Arrival Rate ($\lambda_{\text{target}}$): What the test scenario requested.
2. Achieved Request Rate ($\lambda_{\text{achieved}}$): What the load generator actually dispatched (accounting for dropped_iterations or VU exhaustion).
3. Goodput ($\lambda_{\text{good}}$): Requests that succeeded within the SLO latency bound and passed business correctness checks.

3. Correlate Client Symptoms with Server USE & RED Metrics

Pair external RED metrics (Rate, Errors, Duration) with internal USE Method (Brendan Gregg: Utilization, Saturation, Errors) across every tier:

• CPU & Runtime: CPU utilization, run-queue saturation / container CPU throttling (container_cpu_cfs_throttled_seconds_total), Node.js event-loop lag (perf_hooks.monitorEventLoopDelay), Go goroutine count, JVM/Python/V8 GC pause duration and frequency.
• Memory: RSS vs. heap usage, allocation rate, swap/page faults, OOM kills, and whether memory returns to baseline after traffic stops.
• Network & Pools: Active vs. idle DB/HTTP/Redis connections, pool wait-queue depth, pool acquisition timeouts, ephemeral port exhaustion (TIME_WAIT), and upstream rate limits.
• Disk & Storage I/O: IOPS, read/write await latency, and disk queue depth.

4. Universal Scalability Law (Neil Gunther) & Bottleneck Identification

Never extrapolate linear scaling ("1 instance handles 500 RPS, so 10 instances will handle 5,000 RPS") without measurement. Under Gunther's Universal Scalability Law: $$C(N) = \frac{N}{1 + \alpha(N - 1) + \beta N(N - 1)}$$

• Contention ($\alpha$): Serialization on shared resources (database connection pool limits, row/table locks, shared Redis keys, single-threaded bottleneck) flattens throughput to a ceiling.
• Coherency / Coordination ($\beta$): Cross-node lock contention, distributed cache invalidation storms, or retry storms cause retrograde scaling (adding concurrency or nodes actually decreases total goodput).

5. Falsifiable Root-Cause Evidence

• Never claim a root cause from correlation alone. State:
  1. Measured Fact: Exact metric values, timestamps, environment, and load level where the knee occurred.
  2. Bottleneck Hypothesis: Which specific resource saturated first and what evidence would falsify that explanation.
  3. Unverified Assumptions: What differs between the test environment (mocked third parties, smaller DB volume, single node) and production.