AI streaming performance

AI & Streaming Performance

When to Use

Use when load testing or benchmarking LLM-powered endpoints, SSE/WebSocket streaming responses, multi-step AI agent runtimes, vector search / RAG pipelines, or tool-calling workflows.

Separate Simulated Provider Load from Live Model Calls

Live LLM calls are slow, non-deterministic, rate-limited, and expensive. Always separate AI performance testing into two distinct modes:

1. Deterministic Mock / Replay Upstream Mode (Gateway & Runtime Stress):
   • Use a local mock LLM server that simulates realistic Time to First Token (TTFT) (e.g., 300ms–1500ms), Inter-Token Latency (ITL / TPOT) (e.g., 15–40ms per SSE chunk), tool-call payloads, and occasional 429/529 errors.
   • Use this mode to stress-test the application's own SSE/WebSocket proxy buffering, DB state persistence, event loop, memory footprint per open stream, and concurrency limits at high RPS with zero token cost.
2. Bounded Live Provider Benchmark Mode (End-to-End Calibration):
   • Run strictly bounded concurrency and request counts (with a hard token/cost cap) against the real provider to measure actual model latency distributions, prompt cache hit rates, and RPM/TPM rate-limit behavior.

Streaming & LLM-Specific SLIs to Measure

Standard HTTP http_req_duration only measures total stream completion time—which hides frozen streams! Always measure:

1. TTFT (Time to First Token / First Meaningful Chunk): Time from request dispatch until the first content token or tool event arrives (p50, p95, p99).
2. TPOT / ITL (Time Per Output Token / Inter-Token Latency) and Token Throughput (tokens/sec): Detects stream stuttering or reverse-proxy buffering (e.g., Nginx/Cloudflare buffering the entire SSE stream instead of flushing chunks immediately).
3. Total Completion Time & End-to-End Agent Turn Latency: Breakdown across prompt assembly / RAG retrieval $\rightarrow$ LLM inference $\rightarrow$ tool execution $\rightarrow$ state persistence.
4. Stream Interruption & Client Abort Cleanup:
   • Simulate clients disconnecting mid-stream (AbortController.abort() after 2s).
   • Verify the backend immediately cancels the upstream LLM stream and aborts in-flight tool subprocesses rather than continuing to burn tokens and compute in the background.

Provider Quotas, Retries & Token Cost Economics

• Rate-Limit Handling (429 RPM / TPM / ITPM / OTPM):
  • Verify that concurrent agent runs respect Retry-After / x-ratelimit-reset-* headers, use token-aware queuing or semaphores, and fail over or degrade gracefully without triggering retry storms.
• Cost & Token Accounting Under Load:
  • Track input tokens, cached input tokens (cache_read_input_tokens), output tokens, and total cost per workflow run: $$\text{Cost per Run} = \sum_{\text{steps}} \left( T_{\text{in}} \cdot C_{\text{in}} + T_{\text{cached}} \cdot C_{\text{cached}} + T_{\text{out}} \cdot C_{\text{out}} \right)$$
  • Detect runaway agent loops, unbounded context growth across turns, or missing prompt-cache hits that multiply latency and cost under concurrent load.