Skip to main content

Usage

CPU Run​

Run the image via Docker without options to measure the audio and CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary video scenarios:

docker run --rm ghcr.io/sparecores/benchmark-ffmpeg:main > results.json

The four GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary scenarios are skipped with skip_reason no_cuda_gpu_detected. No environment variables are required, and the image needs no network access at run time: the test media is downloaded and checksum-verified when the image is built.

The --version option prints the versions of the harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary (benchmark.py) and FFmpegFFmpegA free, open-source collection of libraries and command-line tools to decode, encode, transcode, filter and stream audio and video. Its ffmpeg command converts between nearly all media formats, and its libraries, such as libavcodec, power many media players and services.`ffmpeg` docsGlossary, and exits:

docker run --rm ghcr.io/sparecores/benchmark-ffmpeg:main --version

GPU Run​

Pass the GPUs to the container to add the NVIDIA video scenarios:

docker run --rm --gpus all ghcr.io/sparecores/benchmark-ffmpeg:main > results.json

Before running, check the following prerequisites:

  • The host has an NVIDIA GPU, the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary, and the NVIDIA Container Toolkit. The image does not include the CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture) toolkit or NVIDIA driver libraries. It loads NVENCNVENC: NVIDIA Video EncoderThe hardware video encoder of NVIDIA GPUs, a dedicated unit next to the CUDA cores. It encodes H.264, HEVC and, on newer GPUs, AV1, much faster than software encoders, though usually at a somewhat lower quality for the same bitrate. FFmpeg exposes it as encoders such as h264_nvenc.NVIDIA GPUs in FFmpegGlossary (NVIDIA Video Encoder) and NVDECNVDEC: NVIDIA Video DecoderThe hardware video decoder of NVIDIA GPUs, a dedicated unit next to the CUDA cores. It decodes codecs such as H.264, HEVC, VP9 and, on newer GPUs, AV1, leaving the CPU and the CUDA cores free.NVIDIA GPUs in FFmpegGlossary (NVIDIA Video Decoder) from the host driver at run time.
  • nvidia-smi works inside the container. The harness uses it to find the GPUs; without it, the GPU scenarios are skipped.

The image sets NVIDIA_DRIVER_CAPABILITIES=compute,utility,video, so the NVIDIA Container ToolkitNVIDIA Container ToolkitA set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container.NVIDIA docsGlossary also mounts the driver's video libraries without a separate request. Without the video capability, the GPU scenarios fail to load libnvcuvid.so.1 and are skipped.

Each GPU scenario first runs a one-worker probe. If the GPU cannot open an NVENC or NVDEC session for it, the scenario is skipped with the error text as its skip_reason.

Container Resource Limits​

This benchmark sets its worker counts based on the resources available to the container, as reported by cgroup v2cgroup v2: control groups v2The Linux kernel feature that organizes processes into a hierarchy of groups, and limits and accounts for the CPU, memory and I/O of each group. Container runtimes such as Docker use it to enforce the resource limits of a container. It replaced the original cgroup v1 with a single, unified hierarchy.Linux kernel docsDocker resource constraintsGlossary (control groups v2) and CPU affinityCPU affinityThe set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread binding or CPU pinning.Linux man pageGlossary. The following Docker resource limits all lower the worker counts:

  • --cpus (cgroup cpu.max)
  • --cpuset-cpus (CPU affinity)
  • --memory (memory.max): each audio worker is budgeted 64 MiB and each video worker 320 MiB of available memory.
  • --pids-limit (pids.max): each worker is budgeted FFMPEG_BENCH_PID_TASKS_PER_WORKER tasks (default: four), because the PIDPID: Process IdentifierThe number by which the operating system identifies a running process. On Linux, a PID is assigned when a process starts and can be reused after it exits, and commands such as kill use it to target a process.WikipediaGlossary (Process Identifier) limit counts threads, not only processes.

The output records the limits it found in host, including cgroup_cpu_max, cgroup_cpuset_effective, and the cgroup_cpu_stat_before and cgroup_cpu_stat_after snapshots. The _gb memory fields are in GiB.

The harness copies the test media into a work directory on /dev/shm when it has at least 512 MiB free, and into the default temporary directory otherwise. Docker's default /dev/shm is 64 MiB, so a plain docker run uses /tmp.

Worker Counts​

Each scenario is measured at several worker counts (parallel ffmpeg processes), set in benchmark.py as follows:

  • CPU scenarios: 1, P/2 (rounded down), and P, where P is the number of physical cores the container can use. Logical CPUs that are SMTSMT: Simultaneous MultithreadingA CPU technique that runs two (or more) hardware threads on one physical core, sharing its execution units, e.g. Intel Hyper-Threading. It raises throughput when a thread stalls, but the threads compete for the core. Cloud vendors usually sell each thread as a vCPU.WikipediaGlossary (Simultaneous Multithreading) siblings of the same core count once. If throughput at P is at least 1.08 times the throughput at P/2, one more point at 2·P is added, capped by the logical CPU count and the limits above.
  • GPU scenarios: 1 and G, where G is the number of GPUs. The search then keeps doubling the worker count (2·G, 4·G, ...) as long as throughput does not drop, up to eight sessions per GPU by default. Any drop stops the search; the 1.08 ratio does not apply to GPUs.

A worker count with a failed worker stops the search for that scenario. On servers with more than one NUMANUMA: Non-Uniform Memory AccessA memory design where each CPU socket or chiplet has its own local memory, which its cores reach faster than the memory of the others. Software that keeps its threads and their memory on the same NUMA node avoids the slower remote accesses.Linux kernel docsGlossary (Non-Uniform Memory Access) node, workers are spread across the nodes in proportion to their CPUs and bound to them with numactl, if binding works in the container. Otherwise, normal Linux scheduling applies.

For example, a server with 48 physical cores and 96 logical CPUs, with no container limits, measures the CPU scenarios at 1, 24, and 48 workers, and at 96 workers if 48 workers beat 24 by at least 8%.

The following settings change the worker counts:

  • FFMPEG_BENCH_WORKERS replaces the automatic counts with an explicit, comma-separated list, capped at the maximum worker count, and disables the search.
  • FFMPEG_BENCH_OVERSUBSCRIPTION above 1 adds one point at that multiple of P (or of G, for GPU scenarios).
  • FFMPEG_BENCH_MAX_WORKERS caps the worker count.

Run Duration​

Each scenario runs the following steps:

  1. A one-worker probe with 1 second of video or 5 seconds of audio checks that the scenario can run and estimates its speed. A scenario whose probe fails is skipped.
  2. At each worker count, a pilot run sizes the media for each worker so that one repetition takes about FFMPEG_BENCH_TARGET_SECONDS (default: 5s) of wall time. Faster servers therefore process more media in the same time. A successful pilot is not reported; a pilot that still fails after up to four attempts is reported as a failed repetition and ends the scenario.
  3. By default, three repetitions are measured. While the CVCV: Coefficient of VariationThe standard deviation of a set of measurements divided by their mean, often given as a percentage. As it is scale-free, it compares the variability of measurements of different magnitudes, such as how much repeated runs of different benchmarks vary.WikipediaGlossary (Coefficient of Variation) of their rates is above 10%, more are added, up to five. A repetition with a failed worker ends the worker count.

Each repetition, including pilots, times out after FFMPEG_BENCH_REPETITION_TIMEOUT_SECONDS (default: 30s), and workers that run past it count as failed. The whole run stops after FFMPEG_BENCH_TIMEOUT_SECONDS (default: 7,200s); scenarios that have not started by then are reported as skipped with overall_timeout.

Total run time therefore depends on the number of scenarios that run and the worker counts each one measures. A production run on an AWSAWS: Amazon Web ServicesThe cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning.AWSGlossary (Amazon Web Services) m9g.16xlarge with 64 vCPUsvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary and no GPUs measured every CPU and audio scenario at 1, 32, and 64 workers and took about 10 minutes (recorded output).

Across 372 production runs of harness version 3.2.1, run time ranged from about 3 minutes on servers with 2 vCPUs to about 17 minutes on GPU servers, with a median of about 9.5 minutes.

Repetition Length​

Each repetition targets 5 seconds of wall time by default. Because the pilot scales the media to that target, larger servers process more media in the same time instead of running longer.

The default comes from a study that compared targets of 2, 5, and 10 seconds on one developer workstation, a 24-core Intel Core i9-14900K desktop CPU without an NVIDIA GPU, so GPU scenarios were not part of it. Only the study's summary was kept; see Why 5 Seconds.

Shorter Test Runs​

For a quick check that the image works, reduce the worker counts, repetitions, and media lengths:

docker run --rm \
-e FFMPEG_BENCH_WORKERS=1,2 \
-e FFMPEG_BENCH_REPETITIONS=1 \
-e FFMPEG_BENCH_TARGET_SECONDS=1 \
-e FFMPEG_BENCH_MIN_MEDIA_SECONDS=1 \
-e FFMPEG_BENCH_MAX_MEDIA_SECONDS=5 \
-e FFMPEG_BENCH_AUDIO_CALIBRATION_SECONDS=1 \
ghcr.io/sparecores/benchmark-ffmpeg:main

The results of such a run are not comparable with default runs.

Production Setup​

The Inspector of Spare Cores Navigator runs this image on every server type except those on the inspector_rules.py blocklist, with the settings in the following table. The last column shows what a plain docker run does instead:

SettingProductionPlain docker run
Privileges and networkingPrivileged, with host networkingNeither
--shm-sizeNot setNot set
GPUsRequested for the container, equivalent to --gpus all; the run is retried without GPUs if that failsNone, unless you pass --gpus all
Task timeout2 hours2 hours, the harness default
Minimum memoryServers with less than 1 GiB of memory are skippedNot enforced by the image

This benchmark does not need privileged mode or host networking. With no --shm-size set, /dev/shm keeps Docker's 64 MiB default and the test media is copied to /tmp, in production and in a plain run.

We investigate failed production runs, manually or semi-automatically, and then run them again. Until then, a server type's latest result in sc-inspector-data can be a failed or partial run.

Environment Variables​

The following variables configure this benchmark. See Results for the output fields.

VariableMeaningDefault
FFMPEG_BENCH_TARGET_SECONDSTarget wall time per repetition in seconds.5
FFMPEG_BENCH_MIN_MEDIA_SECONDSMinimum media length per worker in seconds.0.5
FFMPEG_BENCH_MAX_MEDIA_SECONDSMaximum media length per worker in seconds.1800
FFMPEG_BENCH_VIDEO_CALIBRATION_SECONDSMedia length of the one-worker video probe in seconds.1
FFMPEG_BENCH_AUDIO_CALIBRATION_SECONDSMedia length of the one-worker audio probe in seconds.5
FFMPEG_BENCH_REPETITIONSMeasured repetitions per worker count (at least 1).3
FFMPEG_BENCH_MAX_REPETITIONSMaximum repetitions per worker count when the rates vary.5
FFMPEG_BENCH_CV_THRESHOLDAdd repetitions while the coefficient of variation of the rates is above this value.0.10
FFMPEG_BENCH_OVERSUBSCRIPTIONAbove 1, adds a worker count at this multiple of the physical cores (CPU) or GPUs (GPU).1
FFMPEG_BENCH_SCALE_CONTINUE_RATIOCPU only: throughput gain from P/2 to P workers required to try 2·P workers.1.08
FFMPEG_BENCH_WORKERSComma-separated worker counts; replaces the automatic counts and disables the search.Automatic
FFMPEG_BENCH_MAX_WORKERSHard cap on the worker count.Automatic
FFMPEG_BENCH_GPU_ENCODE_SESSIONS_PER_GPUHighest NVENC worker count per GPU the search can reach.8
FFMPEG_BENCH_GPU_DECODE_SESSIONS_PER_GPUHighest NVDEC worker count per GPU the search can reach.8
FFMPEG_BENCH_PID_TASKS_PER_WORKERTasks budgeted per worker against the container's PID limit (at least 2).4
FFMPEG_BENCH_REPETITION_TIMEOUT_SECONDSTimeout for each repetition or pilot in seconds.6 × target, at least 30
FFMPEG_BENCH_TIMEOUT_SECONDSDeadline for the whole run in seconds.7200
FFMPEG_BENCH_AUDIO_SOURCEPath of an alternate FLAC file. It is not checksum-verified.Bundled sample
FFMPEG_BENCH_VIDEO_SOURCEPath of an alternate H.264H.264A lossy video codec standardized in 2003, also called AVC. It is the most widely supported video format, used in Blu-ray, broadcast, streaming, and video calls. Common encoders are x264 and hardware encoders such as NVENC.SpecificationGlossary MP4MP4: MPEG-4 Part 14A container file format that holds video, audio, and subtitle tracks, most often H.264 or HEVC video with AAC audio. It is the common format for video on the web and on phones.WikipediaGlossary file. It must match FFMPEG_BENCH_VIDEO_SHA256.Bundled clip
FFMPEG_BENCH_VIDEO_HEVC_SOURCEPath of an alternate H.265H.265Another name for HEVC, from the number of its standard.SpecificationGlossary MP4 file. It must match FFMPEG_BENCH_VIDEO_HEVC_SHA256.Bundled clip
FFMPEG_BENCH_VIDEO_SHA256Expected SHA-256SHA-256: Secure Hash Algorithm 256A cryptographic hash function of the SHA-2 family that maps any data to a 256-bit digest. Even a one-bit change in the data gives a completely different digest, so it is used to verify that a downloaded file is intact, and in digital signatures and certificates.NIST FIPS 180-4Glossary (Secure Hash Algorithm 256) hash of the H.264 file; empty skips the check.Pinned hash
FFMPEG_BENCH_VIDEO_HEVC_SHA256Expected SHA-256 of the H.265 file; empty skips the check.Pinned hash
FFMPEG_BENCH_CDN_BASE_URLBase URLURL: Uniform Resource LocatorThe address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.Glossary recorded as the test media source in the output; nothing is downloaded.https://cdn.sparecores.net/sc-inspector/benchmark-ffmpeg
FFMPEG, FFPROBEAlternate ffmpeg and ffprobe binaries.ffmpeg and ffprobe on PATH