Resumable multipart uploads

Multipart & Resumable Uploads

Single-request uploads (PUT or multipart/form-data) become fragile above 25–100 MB on mobile, Wi-Fi or international connections because a brief network drop forces the user to restart from 0%. Use chunked/multipart or resumable protocols for large files so only the interrupted chunk is retried.

1. S3-Compatible Multipart Upload Lifecycle

Follow the canonical server-coordinated S3 Multipart flow:

1. Initiate (CreateMultipartUpload):
   • Server validates user permissions and quota, creates a local upload_sessions row (status: 'active', owner_id, tenant_id, storage_key, upload_id, declared_size, part_size, expires_at), and calls CreateMultipartUpload.
   • Part sizing rule: S3 requires every part except the last to be at least 5 MiB (5,242,880 bytes), with at most 10,000 parts (1..10000). Choose a dynamic part size (e.g., 8 MiB–64 MiB) based on file size.
2. Sign Parts (UploadPart):
   • Issue short-lived presigned UploadPart URLs per part number (or batch of parts), verifying on every request that the authenticated user owns the upload_session.
   • Include Content-MD5 or x-amz-checksum-sha256 per part where supported to detect corruption in transit.
3. Resume After Interruption (ListParts):
   • When a client reconnects after a drop or page reload, query the session's already-uploaded parts (ListParts or local part table) so the client resumes from the first missing part.
4. Complete (CompleteMultipartUpload):
   • Client sends the ordered list of { PartNumber, ETag }.
   • Server verifies part count and total size (HeadObject), calls CompleteMultipartUpload, and transitions the file to 'uploaded' / 'processing'.
   • Important: in S3, a completed multipart object's ETag is <md5-of-part-md5s>-<partCount> (e.g., "a1b2c3...-4"), not the MD5 of the full file! Use x-amz-checksum-sha256 or stream hashing if a full-file cryptographic digest is needed.
5. Abort & Bucket Lifecycle Cleanup (AbortMultipartUpload):
   • Incomplete multipart parts still consume billable storage until aborted!
   • Call AbortMultipartUpload when a user cancels, and configure both an application cleanup job and an S3 Bucket Lifecycle Rule (AbortIncompleteMultipartUpload after 1–7 days).

2. Resumable Upload Protocols (tus / Content-Range)

When uploads stream through your own server or edge worker (e.g., tus v1.0.0 protocol or HTTP PATCH with Upload-Offset):

• Strict offset verification: before appending bytes on PATCH, check that the client's Upload-Offset matches the server's current persisted byte offset (409 Conflict on mismatch so the client queries HEAD and resumes from the true offset).
• Concurrency lock: lock the upload session row/file handle during an active PATCH stream so two parallel tabs or retry loops cannot interleave chunks into the same file.
• Checksum & max-size enforcement: reject chunks that push current_offset + chunk_length > Upload-Length immediately.