File lifecycle architecture

File Lifecycle & Storage Architecture

Reliable file and media handling separates storage bytes (the object in S3/Blob/disk), file metadata & state (the database record), derived assets (thumbnails, transcodes, previews) and access authorization (who may read or mutate the file). Never equate a storage path or filename with ownership or readiness.

1. Explicit File Lifecycle State Machine

A file is not ready the moment a database row is created or a presigned URL is issued. Model the file lifecycle with explicit states:

1. initiated / uploading: upload session created and quota reserved, bytes are still in transit to temporary/staging storage.
2. uploaded / verifying: client signaled completion; server is verifying actual object existence, byte size, checksum/digest, magic-byte content type and malware/quarantine status.
3. processing: background workers are generating required derivatives (thumbnails, EXIF stripping, waveform/video transcode, document preview).
4. ready (or partially_ready): primary file and required derivatives are verified and available for authorized download/preview.
5. failed / rejected: validation, virus scan or processing failed; reason recorded, quota released, and untrusted bytes scheduled for purge.
6. archived -> soft_deleted -> purging -> purged: logical deletion revokes user access immediately while background cleanup removes primary bytes, all derived variants, multipart fragments and CDN cache entries after any retention grace window.

2. Opaque Object Keys & Metadata Separation

• Never use client-supplied filenames as storage keys: user filenames contain path traversal sequences (../../), Unicode normalization quirks, spaces, shell metacharacters, PII and collisions.
• Generate opaque, tenant-partitioned keys:
  • Format: tenants/{tenant_id}/blobs/{yyyy}/{mm}/{uuidv7_or_random_id} (with no user-controlled segments).
  • Store the sanitized original filename (original_filename), declared vs. detected MIME type (content_type), exact size_bytes, cryptographic digest (sha256), storage bucket/provider and owner_id in the database.
• Derived asset lineage (file_derivatives):
  • Track every generated variant (variant_key: 'thumb_256_webp', 'preview_pdf', 'hls_720p'), its storage_key, size_bytes, status, processor_version (so previews can be deterministically rebuilt when pipeline versions change), and a foreign key to the parent file.

3. Quota Reservation -> Commit -> Release

To prevent concurrent uploads from exceeding a user's or workspace's storage quota:

• Reserve on upload initiation: atomically check and reserve declared_size_bytes (reserved_bytes += declared_size_bytes where used_bytes + reserved_bytes + declared_size_bytes <= quota_limit_bytes).
• Enforce hard size bounds at the storage layer: bind content-length-range (or exact Content-Length) into presigned upload policies or stream limiters so a client cannot declare 1 MB and upload 10 GB.
• Commit on completion: verify actual size_bytes via HEAD object metadata, move reserved_bytes into used_bytes, and reject/delete if the uploaded object violates the agreed bounds.
• Release on abort, failure or expiration: when an upload is canceled, fails validation or expires via the abandoned-upload sweeper, release reserved_bytes atomically.