Token- és cache-elszámolás

LLM Token & Cache Accounting

LLM providers report token usage with incompatible semantics. Naive summation across providers routinely double-counts cached tokens or misses hidden reasoning and tool-use tokens by 2x–10x.

Workflow

1. Normalize provider token semantics into disjoint canonical buckets: Map every provider response into disjoint quantities so total cost is a clean dot product of (bucket_quantity * bucket_rate):
   • uncached_input_tokens: Fresh input tokens billed at standard input rate.
   • cache_write_input_tokens: Tokens written into prompt cache (1.25x or 2x base input rate, or TTL-specific write rate).
   • cache_read_input_tokens: Cached input tokens read at discounted rate (0.1x or 0.25x base input rate).
   • standard_output_tokens: Visible completion tokens billed at output rate.
   • reasoning_output_tokens: Internal chain-of-thought / thinking tokens billed at output rate.
2. Handle each provider's exact schema:
   • Anthropic (messages API):
     • usage.input_tokens, usage.cache_creation_input_tokens, and usage.cache_read_input_tokens are mutually exclusive (disjoint).
     • Total prompt context = input_tokens + cache_creation_input_tokens + cache_read_input_tokens.
     • If usage.cache_creation splits ephemeral_5m_input_tokens vs. ephemeral_1h_input_tokens, rate each at its respective write multiplier.
   • OpenAI (chat.completions & responses API):
     • prompt_tokens (input_tokens) already includes prompt_tokens_details.cached_tokens.
     • Compute uncached_input_tokens = prompt_tokens - cached_tokens and cache_read_input_tokens = cached_tokens. Never add cached_tokens on top of prompt_tokens!
     • completion_tokens (output_tokens) already includes completion_tokens_details.reasoning_tokens. Compute standard_output_tokens = completion_tokens - reasoning_tokens.
   • Google Gemini (generateContent usageMetadata):
     • promptTokenCount already includes cachedContentTokenCount. Compute uncached_input_tokens = promptTokenCount - cachedContentTokenCount.
     • thoughtsTokenCount is billed at the output rate; store standard_output_tokens (candidatesTokenCount) and reasoning_output_tokens (thoughtsTokenCount) distinctly.
3. Streaming responses and aborted connections:
   • Enable final usage chunks on streams (stream_options: { include_usage: true } for OpenAI; message_delta / message_stop for Anthropic).
   • If a stream aborts before the final usage frame, record a fallback event with certainty: 'estimated', counting prompt tokens plus tokenized output chunks emitted before abort.
4. Multi-turn agent loops and CLI / subscription modes:
   • Record each turn in an agent loop as its own event linked to runId and turnIndex, tracking contextGrowthTokens across turns to spot quadratic prompt growth or cache-busting prefixes.
   • For subscription CLI runs with no per-call charge, record tokens, duration, turns, and imputedListCostMicros (billingMode: 'subscription_cli' | 'api_metered' | 'byok').