Storage, transfer and compute metering

Storage, Transfer & Compute Metering

Unlike discrete API calls, storage is a stateful gauge over time while network transfer and background compute are continuous or duration-based flows. Metering them accurately requires distinguishing point-in-time snapshots from time-weighted integrals.

Workflow

1. Meter storage using time-weighted integrals (byte-hours / GB-month):
   • Do not bill or attribute monthly storage solely from a single end-of-month snapshot (which can be gamed by deleting files before midnight and misses short-lived multi-terabyte spikes).
   • Maintain a per-tenant storage ledger driven by file lifecycle events (object_created, object_resized, object_deleted) partnered with the file storage specialist:
     • When a tenant's stored bytes change from $B_0$ to $B_1$ at timestamp $t_1$ after holding $B_0$ since $t_0$, accumulate $\Delta \text{byte_seconds} = B_0 \times (t_1 - t_0)$.
     • Periodically emit hourly/daily checkpoint snapshots (byte_hours, peak_bytes, end_of_window_bytes, object_count) to bound recalculation windows.
     • Convert byte-hours to GB-month (or GiB-month) using an explicit, documented denominator (e.g., $10^9 \text{ bytes} \times 730 \text{ hours}$ or actual hours in the billing month).
2. Separate original assets from derived and retained artifacts:
   • Tag storage events by storageCategory:
     • 'primary_upload': User-uploaded files and documents.
     • 'derived_artifact': Generated thumbnails, video transcodes, OCR text indexes, vector embeddings, or build artifacts.
     • 'version_history' / 'soft_deleted_trash': Retained historical versions or soft-deleted objects awaiting GC.
   • This distinction shows whether storage costs are driven by user uploads or internal pipeline amplification.
3. Meter network transfer and bandwidth (ingress, egress, cdn):
   • Track transfer bytes with direction and path: direction: 'ingress' | 'egress_internet' | 'egress_cdn' | 'cross_region_internal'.
   • Attribute signed-URL downloads and CDN edge deliveries back to tenantId and assetId via signed token metadata or access log ingestion.
   • Account for HTTP Range requests (206 Partial Content) using actual bytes transmitted (responseBytesSent) rather than total file size (Content-Length of the full object).
4. Meter compute jobs, sandboxes, and GPU workers:
   • Record wallClockMs, cpuCoreMs, memoryMbSeconds, and gpuMs (with gpuType, e.g., A100-80GB, L4) for background jobs, media transcoders, code execution sandboxes, and export pipelines.
   • Include queue wait time separately from active execution time, and flag zombie or timed-out jobs that consumed maximum worker duration before being killed.