Storage consistency checks

Storage Consistency & Verification

Object storage (S3, GCS, Azure Blob, local disk), the relational database (PostgreSQL, SQLite, MySQL) and background job queues do not share an atomic transaction. Designing for partial failure prevents "ghost metadata" (DB row exists, file missing in storage) and "dark/orphan objects" (bytes sitting in storage forever with no DB row).

1. Ordering Operations Across Storage & Database

Follow Pat Helland's and Martin Kleppmann's patterns for cross-system mutations:

• Upload sequence (Write Metadata Intent First):
  1. Insert the files row in state 'uploading' (recording the target storage_key and reserving quota) before uploading bytes to object storage.
  2. Upload bytes to storage_key.
  3. Verify the object exists in storage (HeadObject: check actual byte size, checksum and storage metadata) and transition the DB row to 'uploaded' / 'processing' -> 'ready'.
  • Why: if the process crashes during step 2, the DB has a traceable 'uploading' record with the exact storage_key so the cleanup worker can delete the orphaned object and release the quota. If you uploaded to storage before writing to the DB, a crash leaves an invisible, untracked object in the bucket forever.
• Deletion sequence (Soft-Delete DB First, Purge Storage Second):
  1. In a DB transaction, mark the file row status = 'soft_deleted' (or 'purging'), revoke access immediately, and enqueue a durable storage-cleanup job.
  2. The cleanup worker deletes all derived variants, multipart uploads and the primary storage object (DeleteObject is idempotent on 404 NoSuchKey), then transitions the DB row to 'purged' (or deletes it) and decrements used_bytes.

2. Idempotent Background Media Processing

Background workers can crash mid-transcode or receive duplicate job deliveries:

• Deterministic derivative keys: write derived outputs to deterministic keys ({file_id}/derivatives/{variant}_{version}.webp) or overwrite atomically so retrying a failed job does not accumulate duplicate orphan files.
• Independent step tracking: record which derivatives succeeded (thumb_sm: ready, preview_pdf: failed) so a retry only re-runs the failed step and the user can still download the original file if a non-critical preview fails.
• Dead-letter handling: after max retries on a corrupt or unsupported file, transition status to 'failed' with a clear diagnostic code rather than looping infinitely.

3. Mandatory Verification Matrix

Before completing any file or media task, verify these concrete scenarios:

1. Happy path end-to-end: upload -> server verification -> derivative generation -> authorized preview/download -> deletion (confirming primary + derivatives are removed and quota is restored).
2. Interrupted / abandoned upload: start an upload and abort mid-stream; verify state handling, retry/resume behavior and quota release.
3. Spoofed & malformed files: upload a .exe or <svg onload=...> renamed to avatar.png, a zero-byte file, and a truncated/corrupt image; verify magic-byte rejection and clean error reporting.
4. Quota boundary & concurrency: trigger parallel uploads when remaining quota only fits one file; verify atomic reservation prevents over-allocation.
5. Cross-tenant access denial: attempt to finalize, download, preview or delete Tenant A's file using Tenant B's credentials; verify 403/404 rejection.