Object storage integration

Object Storage Integration

While AWS S3, Cloudflare R2, MinIO, Backblaze B2, DigitalOcean Spaces, Google Cloud Storage, Azure Blob Storage and local disk all store byte blobs by key, their consistency guarantees, presigned URL capabilities, CORS rules, checksum headers and metadata limits differ. Hide vendor mechanics behind a narrow storage port while respecting provider-specific constraints.

1. Designing the Storage Adapter Boundary

Define a clean internal interface for storage operations without leaking SDK types into business services:

• putStream(key, readableStream, { contentType, contentLength, sha256, metadata })
• getStream(key, { range }) -> { stream, contentLength, contentType, etag }
• head(key) -> { exists, sizeBytes, contentType, etag, checksum, lastModified }
• createPresignedUpload(key, { contentType, contentLengthRange, expiresInSeconds, checksumSha256 })
• createPresignedDownload(key, { expiresInSeconds, responseContentDisposition, responseContentType })
• delete(key) (idempotent — succeeds when object is already absent)
• copy(sourceKey, destKey)

2. Provider-Specific Capabilities & Gotchas

• AWS S3:
  • Block Public Access (BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy, RestrictPublicBuckets) must be enabled on private buckets.
  • Supports both presigned PUT (exact headers bound into signature) and presigned POST (supports ["content-length-range", min, max] policy conditions!).
• Cloudflare R2 / MinIO / S3-Compatible Stores:
  • Verify whether createPresignedPost is supported by the target provider (some S3-compatible providers only support presigned PUT + Content-Length binding).
  • Set region: 'auto' (for R2) or forcePathStyle: true (for local MinIO / custom endpoints) in @aws-sdk/client-s3.
• Azure Blob Storage:
  • Uses Block Blobs (StageBlock -> CommitBlockList for chunked uploads) and SAS (Shared Access Signature) tokens (BlobSASPermissions: r, w, c, d scoped to exact blob path, start/expiry time and https protocol).
• Local Filesystem (Development / Self-Hosted Single-Node):
  • Write uploads to a temporary file on the same filesystem mount as the target directory, then fs.rename() atomically into place once the stream and checksum succeed so readers never observe a half-written file.
  • Shard directories by prefix (blobs/ab/cd/abcdef...) so a single folder never accumulates millions of inodes.

3. Browser CORS & Checksum Headers

Direct browser-to-object-store uploads fail silently if bucket CORS is misconfigured:

• Allow only the application's exact origins (AllowedOrigins), methods (PUT, POST, GET, HEAD), and required headers (Content-Type, Content-MD5, x-amz-checksum-sha256, x-ms-blob-type).
• Expose ETag in ExposeHeaders (Access-Control-Allow-Headers / Access-Control-Expose-Headers: ETag) — without it, browser JavaScript cannot read the ETag response header required to complete S3 Multipart uploads!