Fájltároló migrációja

File Storage Migration

Migrating files between local disk and cloud object storage (or between AWS S3, Cloudflare R2, Azure Blob and GCS, or reorganizing key prefixes) requires moving gigabytes or terabytes of mutable state while live users continue uploading, downloading and deleting files.

1. Store Explicit Location Metadata Per File

To support zero-downtime migration without breaking active files, record the storage location on each files (and file_derivatives) row:

• storage_provider / storage_backend (e.g., 'local', 's3_legacy', 'r2_primary')
• storage_bucket
• storage_key
• sha256 / size_bytes

When each file row knows which backend and key holds its bytes, the application can route reads per-file without guessing or falling back through slow blind 404 probes.

2. Four-Phase Zero-Downtime Migration Pattern

Execute storage migrations in verifiable phases:

1. Phase 1 — Dual-Read / Location-Aware Routing:
   • Deploy read/download/delete handlers that resolve the storage adapter from the file row's storage_backend (or fall back from new_store -> old_store during transition).
2. Phase 2 — Switch New Writes to Target Store:
   • Direct all newly initiated uploads and newly generated derivatives to the target storage backend (storage_backend = 'target').
   • Ensure deletions remove the object from whichever backend(s) currently hold it.
3. Phase 3 — Idempotent Background Backfill & Checksum Verification:
   • Run a resumable batch worker over remaining rows where storage_backend = 'legacy':
     1. Stream object from legacy to target.
     2. Verify target object via HeadObject and compare exact size_bytes and sha256 digest against the source.
     3. Atomically update the DB row (UPDATE files SET storage_backend = 'target', storage_key = :newKey WHERE id = :id AND deleted_at IS NULL).
     4. If the file was soft-deleted while the copy was in flight, delete the newly copied target object immediately so no orphan is left behind!
4. Phase 4 — Verification Audit & Legacy Decommission:
   • Confirm SELECT COUNT(*) FROM files WHERE storage_backend = 'legacy' is 0.
   • Keep the legacy bucket in read-only/archive state for a safety observation window before scheduling final deletion (with explicit authorization).

3. Guardrails During Migration

• Never delete source objects during the copy step: only delete from the legacy store after the target copy is checksum-verified, the DB pointer is committed, and the rollback window has passed.
• Preserve Content-Type and custom metadata when copying objects across buckets so signed downloads do not revert to application/octet-stream.
• Rate-limit backfill workers so migration bandwidth and IOPS do not starve live user uploads or trigger provider 503 SlowDown throttling.