Invisible Video Watermarking at Scale
How Meta built a scalable invisible watermarking system for video content provenance — and why CPUs ended up outperforming GPUs once the actual bottleneck was found.
The assumption is that ML inference belongs on GPUs. Meta’s invisible watermarking system disproved that — not because GPUs are slow, but because the pipeline surrounding the model made GPU utilization structurally impossible to achieve.
Source
Video Invisible Watermarking at Scale
By: Wes Castro, Zeki Yalniz (Meta)
Published: November 4, 2025
Type: Engineering article
Saved because it contains two underreported production realities: that the right hardware choice depends on pipeline topology, not model characteristics alone; and that standard video quality metrics (VMAF, SSIM) fail to catch the specific artifact class that invisible watermarking introduces.
When the pipeline shape beats the hardware
Invisible watermarking is the practice of embedding a signal into media by subtly modifying pixel values — changes imperceptible to a human eye but detectable by software. Unlike a visible logo or a metadata tag, the modification is baked into the image data itself, so it survives re-encoding, cropping, and the various format conversions that happen when content moves across social media. The signal is built with redundancy, meaning partial destruction through editing still leaves enough signal for detection. At Meta’s scale, this serves three concrete use cases: identifying whether a video was AI-generated, establishing which account first uploaded a piece of content, and inferring what device or tool created it.
Earlier watermarking methods from the 1990s used digital signal processing — DCT and DWT transforms — to hide data in an image’s spectral properties. Those worked well in controlled distribution channels but don’t survive the geometric transforms, aggressive re-encoding, and format variety of modern social media. Machine learning approaches, like Meta’s VideoSeal, are substantially more robust against these real-world edits. The tradeoff is compute: running a neural network on every frame of every video is far more expensive than a DCT operation.
The natural infrastructure choice is GPU inference. That’s where Meta started. The team built a custom FFmpeg filter that computes and applies watermark masks to video frames. FFmpeg is the production standard for video processing, and building watermarking as a filter keeps it composable — it drops into existing video pipelines without requiring each pipeline to be rewritten. The filter model is also what made GPU optimization difficult.
An inference service for GPU models works well when models are warmed up and pre-loaded: the service holds the model in GPU memory between requests and pays the loading cost only once. FFmpeg’s filter architecture makes this impossible. Each FFmpeg process loads the model fresh. For large models, loading time is a small fraction of inference time. For the small watermarking model, loading time was a large fraction of total processing time — the GPU was being allocated, loaded, used briefly, and freed, rather than running sustained at high utilization.
The team identified three specific GPU bottlenecks. Data transfer overhead: high-resolution video frames move back and forth between CPU and GPU memory repeatedly, and the bandwidth of this transfer put a ceiling on achievable throughput. Parallel request handling: running multiple watermarking requests simultaneously across GPUs on the same host caused latency to spike dramatically rather than throughput to scale. Model loading cost: even with the model’s small size, the cold-load penalty per FFmpeg process consumed a disproportionate share of end-to-end processing time.
The CPU investigation started with a baseline that looked discouraging: CPU end-to-end performance was more than two times slower than GPU. What changed the outcome was tuning threading parameters for the encoder, decoder, and PyTorch inference separately, and adjusting the frame sampling parameters the watermarking filter used. With those tuned, CPU E2E latency landed within 5% of GPU — and, critically, multiple FFmpeg processes could run in parallel on CPU without degrading per-request latency. That last property is what makes the CPU solution scale: horizontal parallelism on CPU is cheap and predictable, while GPU parallelism for this workload created contention.
The operational efficiency case follows directly. A CPU fleet provisioned for this workload costs less to operate and avoids the specialized scheduling challenges of GPU infrastructure. The load test validated the same result at scale: a pool of CPU workers matched a pool of GPU workers with similar capabilities, at better operational efficiency.
With a viable infrastructure path established, the remaining work was the four-way tradeoff between latency, watermark detection bit accuracy, visual quality, and BD-Rate — the compression efficiency metric, measuring how much the watermark increases the bitrate required to encode the video. These four metrics pull against each other, and improving any one of them costs at least one of the others. A stronger watermark signal improves detection accuracy but introduces more visible artifacts and adds entropy that makes video harder for a codec to compress.
The BD-Rate regression at initial launch was around 20%. At Meta’s video scale, a 20% increase in required bitrate is a real cost to both infrastructure and users on constrained connections. The fix was a frame-selection method: instead of watermarking every frame, the system selects a subset of frames to watermark. Because the watermark signal is built with redundancy, a partial frame coverage still provides sufficient detection accuracy. Watermarking fewer frames adds less entropy per second of video, which reduces the BD-Rate impact without meaningfully degrading detection.
The visual quality problem was harder, because standard metrics didn’t capture it. VMAF and SSIM were designed to measure compression artifacts — blocking, ringing, blurriness. These are the artifacts that video codecs introduce, and the metrics are well-calibrated for them. Invisible watermarking introduces a different artifact class: structured spatial noise that doesn’t look like codec failure. Watermarked frames can score high on VMAF and SSIM while still containing artifacts visible to a trained human eye. The team’s resolution was crowdsourced manual inspection — humans examining frames and flagging artifacts that metrics passed. That subjective evaluation drove the post-processing tuning until the watermark became genuinely invisible. The gap between metric quality and perceptual quality for this artifact type is still an open problem; there’s no programmatic metric that catches it reliably.
My take
The CPU-over-GPU result is a case study in pipeline topology determining infrastructure choice. GPUs are faster at model inference for this model, and that remains true. But inference time is only one term in end-to-end latency for a video processing pipeline. The loading cost, the data transfer cost, the contention from parallelism — all of those are determined by how the inference fits into the surrounding pipeline, not by the raw inference speed. When the surrounding pipeline is FFmpeg, and FFmpeg’s filter model prevents warmed GPU caching, the math changes.
The VMAF/SSIM failure points at a general problem in production ML deployment: validation metrics are calibrated on the distribution of errors that existed when the metric was developed. Invisible watermarking introduces errors outside that distribution. The lesson isn’t that metrics are useless — it’s that any time a new artifact class is introduced into a pipeline, the existing validation suite needs to be checked for coverage over that artifact class, not just applied and trusted.
The four-way tradeoff space — latency, accuracy, quality, BD-Rate — is the kind of design surface that research papers typically optimize along one or two axes. Production deployment has to hold all four simultaneously, and the constraints interact. The frame-selection insight is a practical handle on a problem that the watermarking literature doesn’t fully address, because papers measure watermarks on individual images, not on the encoded video stream that results from applying them.
Where this is useful
- Building ML-based media processing pipelines where the surrounding infrastructure (FFmpeg, GStreamer) constrains the inference serving model — profile the full pipeline before defaulting to GPU.
- Deploying content provenance or integrity systems where detection must survive re-encoding; metadata-based approaches won’t hold up.
- Validating quality in pipelines that introduce novel artifact types — commission manual inspection alongside automated metrics, especially at launch.
- Less applicable where the ML model dominates pipeline runtime and the inference service can maintain warm model state across requests.
Apply this: For any ML model embedded in a media processing pipeline, measure end-to-end latency with the model running cold (freshly loaded per request), not just inference time in isolation. If the cold-load overhead exceeds 20% of total latency, the serving architecture — not the model — is the optimization target.
What I want to remember
Remember: GPU is the right inference hardware when the model can stay warm; when the pipeline forces cold loads, CPU parallelism often wins on end-to-end latency and always wins on operational efficiency.