Skip to main content

Limitations

Disclaimer: What This Benchmark Does Not Deliver​

Output Quality​

This benchmark measures throughput, not the quality of the encoded output. It computes no quality metric, such as PSNRPSNR: Peak Signal-to-Noise RatioA quality metric that compares a processed image or video to its source by the mean squared error of their pixels, in decibels. Higher values mean a closer match. It is simple and widely used, but follows perceived quality only loosely.FFmpeg filterGlossary (Peak Signal-to-Noise Ratio), SSIMSSIM: Structural Similarity Index MeasureA quality metric that compares a processed image or video to its source by local patterns of luminance, contrast and structure. A score of 1 means identical, and the score follows perceived quality more closely than PSNR.FFmpeg filterGlossary (Structural Similarity Index Measure), or VMAFVMAF: Video Multimethod Assessment FusionA perceptual video quality metric by Netflix that combines several elementary metrics with a machine learning model trained on human ratings. It scores from 0 to 100, and is widely used to tune and compare video encoders.GitHubGlossary (Video Multimethod Assessment Fusion).

The encode settings are fixed per scenario, so the quality target stays the same across server types; see CPU and GPU Encode Quality for how CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary and GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary settings relate.

Storage and Output​

Every job sends its encoded output to FFmpegFFmpegA free, open-source collection of libraries and command-line tools to decode, encode, transcode, filter and stream audio and video. Its ffmpeg command converts between nearly all media formats, and its libraries, such as libavcodec, power many media players and services.`ffmpeg` docsGlossary's null muxer (-f null -), which discards it. This benchmark therefore does not measure disk, file system, or object-storage writes, and a transcoding service that stores its output can be limited by storage instead.

A Specific Media Pipeline​

All jobs use the same pinned test media and fixed codec settings: a film clip of 30 seconds at 1920x800 pixels and 24 fpsfps: Frames Per SecondThe number of frames, the individual images of an animation or simulation, that a program produces each second. A higher rate means smoother motion or faster processing.WikipediaGlossary (Frames Per Second), and a music sample of about 85 seconds. See Test Media for details.

The results do not predict the speed of other resolutions, frame rates, content, codec settings, or codecs, such as AV1AV1: AOMedia Video 1A royalty-free video codec by the Alliance for Open Media, the successor of VP9. It compresses better than H.264 and HEVC at the same quality, at the cost of slower encoding. Common encoders are libaom, SVT-AV1 and rav1e.SpecificationGlossary (AOMedia Video 1). They give a relative sense of how much of this fixed work each server type completes.

Single-Job Speed​

This benchmark measures the combined throughput of many parallel jobs, each limited to one codec thread. The one-worker step shows the speed of a single such job, not of one ffmpeg process that spreads a single stream across many threads. See One Codec Thread per Job.

Scope Decisions​

One Codec Thread per Job​

Each worker is an independent ffmpeg process limited to one codec thread (benchmark.py). The libx265 encoder additionally runs with one frame thread and no thread pool. Throughput on larger servers comes from running more workers in parallel, not from more threads per job.

Physical Cores, Not Logical CPUs​

The CPU worker counts are based on physical cores rather than logical CPUs. Two single-threaded workers on the two hyperthreads of one core compete for that core, so counting logical CPUs over-subscribes servers with SMTSMT: Simultaneous MultithreadingA CPU technique that runs two (or more) hardware threads on one physical core, sharing its execution units, e.g. Intel Hyper-Threading. It raises throughput when a thread stalls, but the threads compete for the core. Cloud vendors usually sell each thread as a vCPU.WikipediaGlossary (Simultaneous Multithreading) and can lower throughput, especially for decoding.

When throughput at the physical core count is at least 8% higher than at half of it, one extra measurement at twice that count shows whether SMT helps; see Worker Counts.

Audio on the CPU Only​

FFmpegFFmpegA free, open-source collection of libraries and command-line tools to decode, encode, transcode, filter and stream audio and video. Its ffmpeg command converts between nearly all media formats, and its libraries, such as libavcodec, power many media players and services.`ffmpeg` docsGlossary's NVIDIA encoders and decoders handle video only, and there is no NVIDIA hardware encoder for VorbisVorbisA royalty-free, lossy audio codec by the Xiph.Org Foundation, usually stored in Ogg files. It was designed as a free alternative to MP3 and AAC, and is used in games and streaming; Opus has largely replaced it for new uses.VorbisGlossary, OpusOpus: audio codecA royalty-free, lossy audio codec standardized by the IETF in 2012. It covers speech and music from low to high bitrates with low delay, which makes it common in voice and video calls and web streaming.OpusGlossary (audio codec), AACAAC: Advanced Audio CodingA lossy audio codec standardized in MPEG-2 and MPEG-4 as the successor of MP3. It sounds better than MP3 at the same bitrate, and it is the usual audio track of MP4 files and streaming video.FFmpeg AAC encoderGlossary (Advanced Audio Coding), MP3MP3: MPEG-1 Audio Layer IIIA lossy audio codec and file format from the 1990s, still the most widely supported format for compressed music.WikipediaGlossary, or FLACFLAC: Free Lossless Audio CodecA lossless audio codec by the Xiph.Org Foundation. It compresses audio without discarding any information, so decoding restores the original samples bit for bit.FLACGlossary (Free Lossless Audio Codec).

The audio scenarios therefore always run on the CPU, and a GPU does not change the audio results. The output records this as audio_gpu_acceleration: false. See the NVIDIA FFmpeg guide for the supported video codecs.

NVIDIA GPUs Only​

The FFmpeg build in this image enables only NVIDIA's video acceleration: NVENCNVENC: NVIDIA Video EncoderThe hardware video encoder of NVIDIA GPUs, a dedicated unit next to the CUDA cores. It encodes H.264, HEVC and, on newer GPUs, AV1, much faster than software encoders, though usually at a somewhat lower quality for the same bitrate. FFmpeg exposes it as encoders such as h264_nvenc.NVIDIA GPUs in FFmpegGlossary (NVIDIA Video Encoder) and NVDECNVDEC: NVIDIA Video DecoderThe hardware video decoder of NVIDIA GPUs, a dedicated unit next to the CUDA cores. It decodes codecs such as H.264, HEVC, VP9 and, on newer GPUs, AV1, leaving the CPU and the CUDA cores free.NVIDIA GPUs in FFmpegGlossary (NVIDIA Video Decoder). It enables no other GPU backend, so other vendors' GPUs are not used. Within NVIDIA, the build targets datacenter GPUs; Jetson devices are not supported.

Raw Results Only​

The output contains every measured repetition, but no headline score, peak, or summary statistic. Aggregate the repetitions yourself; see Results.

Design Constraints​

CPU and GPU Encode Quality​

NVENC has no CRFCRF: Constant Rate FactorThe quality-based rate control of the x264 and x265 encoders, which varies the bitrate to keep the perceived quality constant across scenes. Lower values mean higher quality and larger files; x264's default is 23 on a scale of 0 to 51.FFmpeg wikiGlossary (Constant Rate Factor), the quality setting the CPU encodes use. The scenarios use the following settings:

  • CPU encodes: libx264 and libx265 with -crf 18 and the encoders' default preset (medium).
  • GPU encodes: h264_nvenc and hevc_nvenc with -preset p4 -rc vbr -cq 22 -b:v 0, a CQCQ: Constant QualityThe quality-based rate control of NVIDIA's NVENC hardware encoder, which varies the bitrate to keep a target quality level. Lower values mean higher quality and larger files, as with CRF in software encoders.NVENC rate controlGlossary (Constant Quality) mode.

A sweep of NVENC quality levels on the test clip found that -cq 22 matched libx264 -crf 18 most closely, at an SSIM to the source of about 0.9937 and a bitrate of about 4 Mbps. The match is approximate, not exact, so compare CPU and GPU encode rates with that in mind. See the NVENC CQ sweep for the data.

GPU Coverage​

Before each scenario, a one-worker probe checks that it can run. A scenario whose probe fails is skipped with the error text as its skip_reason. This happens, for example, when a GPU cannot open an NVENC or NVDEC session. A missing GPU result therefore means the scenario could not run in that environment.

GPU scenarios also need the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary's video libraries inside the container. Since harness version 3.3.0, the image requests them itself; earlier production runs on many GPU servers skipped all GPU scenarios because these libraries were missing.

Production results for GPUs are therefore incomplete until those servers are measured again.

Computed Media Counts​

The processed_frames and processed_audio_seconds fields are computed from the media length each worker was asked to process, not read from FFmpegFFmpegA free, open-source collection of libraries and command-line tools to decode, encode, transcode, filter and stream audio and video. Its ffmpeg command converts between nearly all media formats, and its libraries, such as libavcodec, power many media players and services.`ffmpeg` docsGlossary's output: successful workers × media length (× the clip's frame rate, for video). A failed or timed-out worker adds no media, but the group's wall time still includes it, so failures lower the rate.