Capacity planning and reserve

Capacity Planning

When to Use

Use when translating load test measurements into concrete capacity limits, autoscaling thresholds, hardware/instance sizing, or cost-efficiency models.

Defining Capacity Rigorously

"Concurrent users" or "CPU at 60%" is never a capacity statement on its own. A valid capacity statement always binds four elements together: $$\text{Capacity} = \text{Max Sustainable Arrival Rate } (\lambda_{\text{max}}) \text{ for Workload Mix } W \text{ meeting SLOs } S \text{ on Environment } E$$ Example: "On 2 app instances (2 vCPU / 4 GB RAM) + Postgres db.m6i.large with 1M seeded orders, the system sustains 320 RPS (70% read / 20% search / 10% write) with p95 = 185ms < 250ms, p99 = 410ms < 500ms, and error rate = 0.02% < 0.1%."

Finding the Knee of the Curve & Safe Operating Headroom

1. Locate the Saturation Knee ($\lambda_{\text{knee}}$)

Using step-ramp load tests:

• Plot Achieved Goodput and p95 / p99 Latency against Arrival Rate ($\lambda$).
• Identify the knee point where latency transitions from flat/linear to steep queueing growth, or where the first resource pool (DB connections, CPU run queue, worker threads, upstream quota) crosses ~75–80% utilization.
• Identify the SLO breach point ($\lambda_{\text{SLO}}$) where p95, p99, or error rate first violates the service level objective.

2. Calculate Required Safety Headroom

Never run production at $\lambda_{\text{SLO}}$:

• In queueing systems ($M/M/1$ / $G/G/c$), average queue wait scales with $\frac{\rho}{1 - \rho}$. At $\rho = 0.70$ utilization, wait time is $2.3\times$ service time; at $\rho = 0.90$, wait time jumps to $9\times$ service time and tiny traffic bursts trigger immediate timeout cascades.
• Recommend a Safe Operating Envelope ($\lambda_{\text{safe}}$) that leaves 25–40% headroom below $\lambda_{\text{SLO}}$, accounting for:
  1. Diurnal peak-to-average ratios and flash bursts.
  2. Autoscaler cold-start and provisioning lag (30s–180s for new containers/nodes to warm up and pass health checks).
  3. $N-1$ redundancy (surviving the loss of one instance or AZ during peak traffic).

3. Map the Bottleneck Chain (First, Second, and External Ceilings)

Document the sequence of limits so engineering effort targets the real constraint:

1. Primary Bottleneck: What saturates first (e.g., Postgres connection pool of 20 connections at 340 RPS).
2. Secondary Bottleneck: What will saturate next once the primary is raised (e.g., CPU bcrypt hashing or Redis single-thread command rate at 600 RPS).
3. Hard External Ceilings: Third-party API rate limits (Stripe, LLM token/RPM quotas, SMTP/SMS provider limits) that cannot be scaled by adding app instances.

4. Unit Economics & Cost-at-Scale

Pair capacity metrics with unit cost:

• Compute resource cost per $10\text{k}$ requests, per active user, or per workflow execution (compute + DB I/O + egress + external API/LLM cost).
• Highlight whether fixing an algorithmic or query bottleneck yields higher headroom than horizontal scaling.