Storage retention and cleanup

Retention & Storage Cleanup

Without deliberate lifecycle cleanup, storage costs grow unchecked from abandoned multipart uploads, failed transcode artifacts, orphaned thumbnails and deleted records whose underlying blobs were left behind. Equally critical: an overly aggressive cleanup script can delete in-flight uploads or violate backup/legal retention rules.

1. Cascading Deletion Completeness

When a user or workspace deletes a file (or an entire project/account is deleted), track every artifact that must be cleaned up:

1. Primary storage object (storage_key).
2. All derived variants (file_derivatives: thumbnails, responsive sizes, PDF previews, audio waveforms, and entire HLS/DASH segment prefixes).
3. Quarantine / staging copies (if deletion happens before promotion).
4. Active multipart upload sessions (AbortMultipartUpload so unmerged parts are freed).
5. Share links & CDN cache: invalidate file_shares immediately and purge edge cache keys if the asset was cached on a CDN.
6. Quota accounting: decrement used_bytes (or release reserved_bytes) atomically once deletion is committed.

2. Two-Phase Deletion & Grace Windows

Never delete storage bytes synchronously inside a user-facing DELETE database transaction (if the DB rolls back after DeleteObject, data is permanently lost):

• Phase 1 — Logical Soft-Delete:
  • Set deleted_at = NOW(), purge_after = NOW() + grace_period (e.g., immediate for temporary drafts, or 7–30 days for recoverable trash / accidental-deletion recovery).
  • Revoke all read/download/share access immediately when deleted_at IS NOT NULL.
• Phase 2 — Asynchronous Physical Purge:
  • A background worker selects rows where status = 'soft_deleted' AND purge_after <= NOW() (and no legal/compliance hold applies), deletes all derivative and primary storage objects, and marks status = 'purged', purged_at = NOW().

3. Safe Orphan Sweeping (Avoiding In-Flight Deletion)

When sweeping for orphaned files or abandoned uploads:

• Enforce a generous minimum age threshold (Safety Grace Window):
  • Never delete an unreferenced storage object or 'uploading' session that is younger than the maximum possible upload/processing duration (e.g., only sweep 'uploading' sessions where expires_at < NOW() - INTERVAL '2 hours', and only sweep unreferenced storage blobs older than 24–72 hours).
• Verify zero active references before deleting:
  • If content-addressed storage or deduplication is used, verify that zero active files rows reference the blob hash (with a lock or reference counter) before deleting the underlying object.
• Dry-run & batch rate limiting:
  • Build mass-cleanup jobs with a --dry-run mode that logs candidate count and total bytes first, and process deletions in bounded batches (DeleteObjects up to 1,000 keys) with audit logging.

4. Object Versioning, Backups & Honest Deletion Claims

• Check whether S3/Blob Object Versioning, Object Lock (WORM) or automated snapshot backups are enabled on the bucket.
• In a versioned bucket, a standard DeleteObject only inserts a Delete Marker while keeping prior versions billable and recoverable unless a VersionId delete or lifecycle expiration rule is configured. Document the actual retention behavior honestly.