Skip to main content

Workloads

Audio Encoding​

Every audio job decodes the bundled FLACFLAC: Free Lossless Audio CodecA lossless audio codec by the Xiph.Org Foundation. It compresses audio without discarding any information, so decoding restores the original samples bit for bit.FLACGlossary (Free Lossless Audio Codec) music sample, resamples it when needed, and encodes stereo output with one codec thread, using VorbisVorbisA royalty-free, lossy audio codec by the Xiph.Org Foundation, usually stored in Ogg files. It was designed as a free alternative to MP3 and AAC, and is used in games and streaming; Opus has largely replaced it for new uses.VorbisGlossary, OpusOpus: audio codecA royalty-free, lossy audio codec standardized by the IETF in 2012. It covers speech and music from low to high bitrates with low delay, which makes it common in voice and video calls and web streaming.OpusGlossary (audio codec), AACAAC: Advanced Audio CodingA lossy audio codec standardized in MPEG-2 and MPEG-4 as the successor of MP3. It sounds better than MP3 at the same bitrate, and it is the usual audio track of MP4 files and streaming video.FFmpeg AAC encoderGlossary (Advanced Audio Coding), MP3MP3: MPEG-1 Audio Layer IIIA lossy audio codec and file format from the 1990s, still the most widely supported format for compressed music.WikipediaGlossary, or FLAC. The audio scenarios run on the CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary only; see Audio on the CPU Only.

ScenarioEncoderSettingsSample rate
ogg_vorbis_160klibvorbis160 kbps44.1 kHz
opus_128klibopus128 kbps48 kHz
aac_128kaac (FFmpegFFmpegA free, open-source collection of libraries and command-line tools to decode, encode, transcode, filter and stream audio and video. Its ffmpeg command converts between nearly all media formats, and its libraries, such as libavcodec, power many media players and services.`ffmpeg` docsGlossary's native AAC encoder)128 kbps44.1 kHz
mp3_192klibmp3lame192 kbps44.1 kHz
flac_losslessflac (FFmpeg's native FLAC encoder)Compression level 544.1 kHz

The opus_128k workload uses 48 kHz because libopus accepts only 8, 12, 16, 24, or 48 kHz, not the sample's 44.1 kHz.

Audio results are reported as audio_seconds_per_sec: seconds of audio processed per second of wall time, across all workers.

CPU Video​

The CPU video scenarios use FFmpegFFmpegA free, open-source collection of libraries and command-line tools to decode, encode, transcode, filter and stream audio and video. Its ffmpeg command converts between nearly all media formats, and its libraries, such as libavcodec, power many media players and services.`ffmpeg` docsGlossary's software codecs, with one codec thread per job for both decoding and encoding:

ScenarioOperationInputDecoderEncoder and settings
cpu_h264_encodeEncodeH.264H.264A lossy video codec standardized in 2003, also called AVC. It is the most widely supported video format, used in Blu-ray, broadcast, streaming, and video calls. Common encoders are x264 and hardware encoders such as NVENC.SpecificationGlossary cliph264libx264 -crf 18
cpu_h265_encodeEncodeH.264 cliph264libx265 -crf 18, one frame thread, no thread pool
cpu_h264_decodeDecodeH.264 cliph264None
cpu_h265_decodeDecodeH.265H.265Another name for HEVC, from the number of its standard.SpecificationGlossary cliphevcNone

The encode scenarios also decode their H.264 input on the CPU, so they measure a full transcode. CRFCRF: Constant Rate FactorThe quality-based rate control of the x264 and x265 encoders, which varies the bitrate to keep the perceived quality constant across scenes. Lower values mean higher quality and larger files; x264's default is 23 on a scale of 0 to 51.FFmpeg wikiGlossary (Constant Rate Factor) 18 is a quality target; no -preset is set, so libx264 and libx265 use their default preset (medium).

Video results are reported as aggregate_fps: frames processed per second of wall time, across all workers.

NVIDIA Video​

The NVIDIA video scenarios decode on the GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary and, for encodes, keep the frames in GPU memory for NVENCNVENC: NVIDIA Video EncoderThe hardware video encoder of NVIDIA GPUs, a dedicated unit next to the CUDA cores. It encodes H.264, HEVC and, on newer GPUs, AV1, much faster than software encoders, though usually at a somewhat lower quality for the same bitrate. FFmpeg exposes it as encoders such as h264_nvenc.NVIDIA GPUs in FFmpegGlossary (NVIDIA Video Encoder):

ScenarioOperationInputDecoderEncoder and settings
gpu_h264_encodeEncodeH.264H.264A lossy video codec standardized in 2003, also called AVC. It is the most widely supported video format, used in Blu-ray, broadcast, streaming, and video calls. Common encoders are x264 and hardware encoders such as NVENC.SpecificationGlossary cliph264_cuvidh264_nvenc -preset p4 -rc vbr -cq 22 -b:v 0
gpu_h264_decodeDecodeH.264 cliph264_cuvidNone
gpu_h265_encodeEncodeH.264 cliph264_cuvidhevc_nvenc -preset p4 -rc vbr -cq 22 -b:v 0
gpu_h265_decodeDecodeH.265H.265Another name for HEVC, from the number of its standard.SpecificationGlossary cliphevc_cuvidNone

Each job runs with -hwaccel cuda on one GPU. The -cq 22 constant-quality setting was chosen to approximately match the CPU encodes at CRF 18; see CPU and GPU Encode Quality.

Test Media​

The image build downloads the test media from the Spare Cores CDNCDN: Content Delivery NetworkA network of distributed servers that cache content, such as web pages, images or video, and serve it from locations close to its users. It lowers latency for the users and offloads traffic from the origin server.WikipediaGlossary (Content Delivery Network) and checks each file's SHA-256SHA-256: Secure Hash Algorithm 256A cryptographic hash function of the SHA-2 family that maps any data to a 256-bit digest. Even a one-bit change in the data gives a completely different digest, so it is used to verify that a downloaded file is intact, and in digital signatures and certificates.NIST FIPS 180-4Glossary (Secure Hash Algorithm 256) checksum. At run time, the H.264H.264A lossy video codec standardized in 2003, also called AVC. It is the most widely supported video format, used in Blu-ray, broadcast, streaming, and video calls. Common encoders are x264 and hardware encoders such as NVENC.SpecificationGlossary and H.265H.265Another name for HEVC, from the number of its standard.SpecificationGlossary clips are checked again; the audio file is not. The files are the following:

  • H.264 clip (source.mp4): an excerpt of 30 seconds from the Blender Foundation's open movie Tears of Steel, cut at 60 seconds from the 1080p release with stream copy. It is 1920x800 pixels at 24 fpsfps: Frames Per SecondThe number of frames, the individual images of an animation or simulation, that a program produces each second. A higher rate means smoother motion or faster processing.WikipediaGlossary (Frames Per Second). License: CC BY 3.0CC BY 3.0: Creative Commons Attribution 3.0A Creative Commons license that allows anyone to copy, adapt and use a work, also commercially, as long as they credit its author.LicenseGlossary (Creative Commons Attribution 3.0), as stated on the Tears of Steel sharing page.
  • H.265 clip (source-hevc.mp4): the same excerpt, transcoded with libx265 -crf 18 -preset medium, so the H.265 decode scenarios get the same frame size, frame rate, and duration.
  • Music sample (source.flac): the "Entre dos Aguas" sample performed by Michael Laucke, about 85 seconds at 44.1 kHz. License: CC0 1.0CC0 1.0: Creative Commons Zero 1.0A Creative Commons dedication with which authors waive all their copyright in a work, placing it as close to the public domain as the law allows. Anyone may use the work for any purpose, without attribution.LicenseGlossary (Creative Commons Zero 1.0).

Each job loops its input (-stream_loop -1) and stops after the media length set for that worker count (-t), so the length of a job does not depend on the length of the file. The harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary (benchmark.py) probes each file's resolution, frame rate, and duration at run time and reports them in sources.

How Measurement Works​

This benchmark runs its 13 scenarios in a fixed order, as defined in benchmark.py: the eight video scenarios first, then the five audio scenarios. Each repetition measures one group of parallel workers in the following steps:

  1. The harness starts every worker as a separate process. Each worker first waits to read from one shared pipe, so ffmpeg has not started yet when the last worker is ready.
  2. The harness records the start time, then closes the pipe. Closing it ends every worker's wait at the same moment, and each worker starts its ffmpeg command. The start time comes from a monotonic clock, which only moves forward and is not changed by system time adjustments such as NTPNTP: Network Time ProtocolA protocol that keeps computer clocks in sync with reference time servers over a network. A client adjusts its system clock in small steps or, when it is far off, in one jump, so wall-clock time on a server can move backward or skip ahead.NTPGlossary (Network Time Protocol) corrections, so the measured wall time cannot jump.
  3. It records each worker's exit time and exit code. The wall time is the time from the release to the last exit, so process startup and teardown are included.
  4. It computes the rate: successful workers × media length per worker, divided by the wall time. For video, the media length is converted to frames with the clip's frame rate.

The harness never adds up the speed that each ffmpeg process reports for itself. Workers that time out are stopped, recorded with exit code 124, and count as failed. Encoded output goes to the null muxernull muxerAn FFmpeg output format, selected with -f null, that discards everything it receives instead of writing a file. It is used to measure decoding, filtering or encoding speed, or to run analysis filters, without the cost of writing to disk.FFmpeg formatsGlossary and is discarded; see Storage and Output.

See Worker Counts for how many workers each group has, and Run Duration for the repetitions and timeouts.