Bounded streaming and processing

Streaming & Resource Limits

Loading a 500 MB upload or video into an in-memory Buffer / byte[] (fs.readFile, res.arrayBuffer(), unbounded multipart buffering) crashes or starves the server as soon as three users upload concurrently. Handle file I/O and media processing with bounded streams, explicit backpressure and strict resource limits.

1. End-to-End Streaming & Backpressure

• Never buffer whole files in memory when streaming is possible:
  • Pipe request/storage streams directly using backpressure-aware primitives (stream.pipeline in Node.js, Web ReadableStream.pipeTo(), Go io.Copy, Python chunked generators, PHP stream resources).
  • Compute cryptographic hashes (SHA-256 / MD5) and inspect initial magic bytes incrementally on the stream via transform streams rather than reading the file twice or buffering it into RAM.
• Direct-to-Storage Bypass:
  • For files larger than a few megabytes when cloud object storage is available, prefer direct client-to-object-storage uploads (presigned URLs / multipart) so application server memory, CPU and event loops are not tied up proxying gigabytes of binary data.
• Range Requests (HTTP 206 Partial Content):
  • Support Accept-Ranges: bytes and Range: bytes=start-end on large downloads, audio/video playback and PDF previews so clients can seek and resume downloads without fetching the entire file.

2. Bounding Processing Memory, CPU, Time & Output Size

Media decoders (libvips/sharp, ImageMagick, ffmpeg, Poppler/pdfium, archive extractors) process complex binary formats that can trigger extreme memory spikes or CPU hangs:

1. Pixel / Dimension Limits (Decompression Bomb Defense):
   • A 400 KB PNG can have dimensions of 60000 x 60000 pixels (3.6 gigapixels), which expands to 14.4 GB of uncompressed RGBA memory when decoded!
   • Inspect image dimensions from the header first and set hard pixel limits (e.g., limitInputPixels in sharp / maxResolution in decoders) before decoding.
2. Execution Timeouts & Abort Signals:
   • Wrap every external CLI (ffmpeg, ffprobe, clamscan) and worker step in a hard timeout (AbortSignal.timeout(ms) / process kill timer) that terminates hung child processes cleanly.
3. Bounded Concurrency & Worker Isolation:
   • Run CPU- and memory-intensive media transformations in background job queues with bounded worker concurrency (concurrency: 2..N matched to container RAM/CPU), never synchronously inside the HTTP request thread.
4. Safe Subprocess Invocation:
   • Spawn external tools (ffmpeg, ffprobe, gs) using argument arrays (execFile / spawn(bin, args, { shell: false })) — never interpolate filenames or metadata into a shell string!
   • When passing paths to ffmpeg or CLI tools, prefix local relative paths (./input.mp4 or absolute paths) so a filename starting with - or http:// is never interpreted as a CLI flag or remote protocol.

3. Ephemeral Scratch Disk Hygiene

• When a tool requires a local file on disk (such as ffmpeg MP4 muxing or virus scanning), write to an isolated temporary directory (mkdtemp), enforce a maximum disk write quota, and guarantee deletion in a try / finally block even when the job throws or times out.