AI-token-, cache- és költségkezelés

Token, Cache & Cost Optimization

In multi-turn agent loops, input tokens grow quadratically ($O(T^2)$) with turn count $T$ because the full conversation history and tool definitions are re-sent on every step. Controlling agent cost and latency requires high prompt-cache hit rates, disciplined tool-output shaping, and right-sized model routing.

Maximizing Prompt Cache Hit Rates

Provider KV-cache reuse (Anthropic cache_control, OpenAI automatic prefix caching, Gemini context caching) cuts input token cost by 75–90% and drastically reduces Time-To-First-Token (TTFT), but a single mutated token near the top of the prompt invalidates the entire downstream cache prefix:

Tool Output & Schema Token Discipline

1. Bound tool outputs at the source: A single unfiltered grep, DOM dump, or log file read can inject 50,000 tokens that get re-billed on every subsequent turn of the run. Enforce strict maxLines / maxBytes, return structured summaries with line numbers, and let the agent fetch targeted slices on demand.
2. Lazy / deferred tool schemas: When an agent has access to dozens of MCP or domain tools, avoid injecting 20,000 tokens of JSON Schema on turn 1 if only 5 core tools are routinely used. Provide a compact catalog or search/discovery tool (defer_loading: true / schema lookup) and load full schemas only when needed.
3. Concise structured intermediate turns: Instruct subagents and internal classification steps to emit compact JSON or terse bulleted findings rather than verbose conversational boilerplate.

Tiered Model Routing & Reasoning Effort

Match model capability and reasoning effort (low | medium | high) to the cognitive complexity of each step:

• Frontier / High-Effort Model: Architecture design, multi-file root-cause diagnosis, subtle concurrency/security review, and complex orchestration planning.
• Mid-Tier / Standard-Effort Model: Routine code implementation following an approved plan, standard tool-calling workflows, and structured test generation.
• Fast / Lightweight Model: Context compaction summaries, query expansion, log/file triage, classification, and schema formatting.

Cost Accounting & Budget Circuit Breakers

Compute exact cost per turn from the provider's normalized usage object: $$\text{Cost}{\text{turn}} = I{\text{uncached}} \cdot P_{\text{in}} + I_{\text{cache_write}} \cdot P_{\text{cw}} + I_{\text{cache_read}} \cdot P_{\text{cr}} + O_{\text{total}} \cdot P_{\text{out}}$$ Track cache_hit_ratio = cache_read_input_tokens / (input_tokens + cache_read_input_tokens + cache_creation_input_tokens) in telemetry. Alert when cache hit ratio drops below expected baselines (typically $> 75%$ on multi-turn agent loops) and enforce hard per-run token and USD ceilings.