Elszámolási események duplikációszűrése

Deduplication & Accounting Integrity

Usage and cost ledgers require financial-grade accounting integrity. Retries, webhook replays, queue redeliveries, and crash recoveries must never inflate usage or silently drop consumption.

Workflow

1. Deterministic idempotency keys:
   • Construct idempotency keys from stable business coordinates rather than random UUIDs generated at logging time:
     • LLM / external API call: {provider}:{providerRequestId} or {runId}:{stepIndex}:{attemptNumber}.
     • Queue job execution: {queueName}:{jobId}:{executionAttempt}.
     • Periodic storage snapshot: {tenantId}:{bucketId}:{windowStartIso}.
   • Enforce a unique constraint (UNIQUE (tenant_id, idempotency_key)) at the database or ingestion layer and handle conflicts with idempotent upserts (ON CONFLICT DO NOTHING or upgrading 'estimated' to 'provider_reported').
2. Append-only immutable ledger:
   • Never UPDATE or DELETE committed usage or cost ledger rows to fix a bug, apply a credit, or change a price.
   • Record corrections, refunds, provider credits, or late invoice adjustments as compensating/adjustment ledger entries (entryType: 'raw_usage' | 'adjustment' | 'credit' | 'reconciliation_delta') referencing originalEventId with a reason code and timestamp.
3. Separate raw events from rollups:
   • Keep the raw event table append-only as the single source of truth.
   • Materialize hourly/daily/monthly rollups (usage_rollups_hourly, usage_rollups_daily) as derived projections that can be deterministically rebuilt from the raw event log.
   • Track a watermark on rollups so late-arriving events trigger incremental recalculation of affected time buckets.
4. Integer and fixed-point arithmetic:
   • Never use IEEE 754 floating-point (number / FLOAT / DOUBLE) to accumulate monetary costs.
   • Store and sum costs in integer sub-cent units (nanodollars 10^-9 USD or microdollars 10^-6 USD using BIGINT / BigInt or NUMERIC(20, 9)) because per-token LLM prices underflow 2- or 4-decimal cent columns.

Anti-Patterns to Avoid

• Random UUID idempotency keys: Generating crypto.randomUUID() inside the logger wrapper makes every retry look like a new event.
• Rounding each event to cents (0.01): Rounding a $0.0004 LLM call before summing causes massive cumulative drift.
• In-place backfills: Overwriting historical ledger rows without an audit trail makes it impossible to explain why past reported cost changed.