Budget reservations

Budget Reservation & Enforcement

Checking if (currentSpend < budgetLimit) before starting an expensive LLM run or batch job is vulnerable to classic Time-of-Check to Time-of-Use (TOCTOU) races: 20 concurrent requests all see $9.50 < $10.00 and together spend $25.00.

Workflow

1. Two-phase budget reservation pattern (Reserve -> Settle / Release):
   • Before launching a costly operation (LLM call, multi-step agent run, batch export), calculate a conservative upper-bound estimate (estimatedCostMicros from prompt token count + max_tokens * output rate).
   • Atomically check and reserve capacity against the budget window: $$\text{committedSpend} + \text{activeReservations} + \text{newReservation} \le \text{hardLimit}$$
   • Persist a reservation record (reservationId, tenantId, budgetId, reservedMicros, status: 'held', expiresAt).
   • When the operation finishes (or fails):
     • Settle: Atomically record the actual ledger entry (actualCostMicros) and transition the reservation to 'settled' (releasing reservedMicros - actualCostMicros).
     • Release on abort: If the operation is rejected before reaching the provider, transition the reservation to 'released'.
2. Lease TTL and crash recovery:
   • Every reservation must carry an expiresAt lease TTL (e.g., 2–10 minutes for interactive calls, heartbeat-extended for long background jobs).
   • A background reaper or lazy expiration query automatically reclaims expired 'held' reservations from crashed workers so a process crash never permanently locks a customer's budget.
3. Multi-level budget scopes and policy actions:
   • Support budgets scoped to tenant, workspace, project, user, apiKey, agentRole, or singleTaskRun (e.g., max $2.00 per task run or max 15 agent turns).
   • Support graduated threshold policies:
     • Soft warning thresholds (50%, 80%, 100%): Emit deduplicated alert events (budget_threshold_crossed keyed by (budgetId, periodStart, thresholdPct)) without interrupting execution.
     • Model downgrade / throttling policy: Automatically route non-critical tasks to a more economical model tier or lower concurrency when approaching the limit.
     • Hard stop (circuit breaker): Reject new reservations with a clear, structured domain error (BUDGET_EXCEEDED) and allow in-flight work to checkpoint cleanly rather than corrupting state mid-write.

Anti-Patterns to Avoid

• Post-hoc-only budget checks: Only checking spend via an hourly cron job allows a runaway agent loop to burn hundreds of dollars in minutes before the next cron tick.
• Unbounded reservation locks: Holding reserved budget without an expiresAt TTL permanently starves a tenant after a worker restart.
• Spamming alert notifications: Firing a webhook on every single API call once spend exceeds 80% instead of recording that the 80% threshold state was already notified for the current period.