Video and audio processing

Video & Audio Processing

Video and audio transcoding are among the most CPU-, disk- and time-intensive operations in a backend system. Design media pipelines around upfront stream probing, safe subprocess execution, streaming-friendly container layouts and resumable background jobs.

1. Upfront Probing & Validation (ffprobe)

Before running an expensive ffmpeg transcode, inspect the container and streams with ffprobe (-v error -show_format -show_streams -of json):

• Verify actual streams: confirm whether the file contains a valid video stream (codec_type === "video"), audio stream (codec_type === "audio"), duration, dimensions (width, height), frame rate and bitrate.
• Enforce media bounds: reject files exceeding maximum duration, resolution or stream count, or files using unsupported/dangerous external reference tracks (such as HLS playlist files containing file:// or internal http:// segments!).
• Disable network protocols on untrusted inputs: pass -protocol_whitelist file,pipe (or local file only) when probing or decoding user-uploaded files so a crafted playlist container cannot trigger SSRF or local file inclusion.

2. Web-Optimized Transcoding & Delivery Formats

• Progressive MP4 (+faststart):
  • When outputting MP4 (H.264/AAC), always pass -movflags +faststart so the moov atom (index table) is moved to the front of the file. Without +faststart, browsers must download the entire multi-hundred-MB file before playback can start!
  • Normalize pixel format (-pix_fmt yuv420p) and audio sample rate/channels (-ac 2 -ar 48000) for universal browser and mobile hardware decoder compatibility.
• Adaptive Bitrate Streaming (HLS / DASH):
  • For longer or high-resolution videos, generate an HLS master playlist (.m3u8) and segmented renditions (360p, 720p, 1080p .ts or fMP4 .m4s segments) so players adapt smoothly to mobile network conditions.
  • Remember that HLS produces many small files (manifest + hundreds of segments): track the rendition prefix cleanly in file_derivatives so storage cleanup deletes the entire segment prefix when the parent video is deleted.
• Poster Frames, Timeline Sprites & Audio Waveforms:
  • Extract a representative poster frame (-ss keyframe + WebP/JPEG output) and, for audio files, precompute normalized peak/RMS waveform JSON arrays during background processing so the UI player renders instantly.

3. Worker Isolation, Progress & Failure Handling

• Never transcode on web/API nodes: run media jobs on dedicated worker queues with bounded concurrency and scratch disk limits.
• Skip redundant transcoding (Stream Copy): if ffprobe shows an uploaded video is already within target resolution/bitrate in h264/aac, remux with -c copy -movflags +faststart in seconds instead of re-encoding from scratch.
• Always clean up scratch files in finally blocks and record clear error classifications (corrupt container vs. missing codec vs. worker timeout).