Usage
CPU Run
Run the image via Docker without options to measure the audio and CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary video scenarios:
docker run --rm ghcr.io/sparecores/benchmark-ffmpeg:main > results.json
The four GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary scenarios are skipped with skip_reason no_cuda_gpu_detected.
No environment variables are required, and the image needs no network access at
run time: the test media is downloaded and checksum-verified when the image is
built.
The --version option prints the versions of the harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary
(benchmark.py) and FFmpegFFmpegA free, open-source collection of libraries and command-line tools to decode, encode, transcode, filter and stream audio and video. Its ffmpeg command converts between nearly all media formats, and its libraries, such as libavcodec, power many media players and services.`ffmpeg` docsGlossary, and exits:
docker run --rm ghcr.io/sparecores/benchmark-ffmpeg:main --version
GPU Run
Pass the GPUs to the container to add the NVIDIA video scenarios:
docker run --rm --gpus all ghcr.io/sparecores/benchmark-ffmpeg:main > results.json
Before running, check the following prerequisites:
- The host has an NVIDIA GPU, the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary, and the
NVIDIA Container Toolkit.
The image does not include the CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture)
toolkit or NVIDIA driver libraries. It loads NVENCNVENC: NVIDIA Video EncoderThe hardware video encoder of NVIDIA GPUs, a dedicated unit next to the CUDA cores. It encodes H.264, HEVC and, on newer GPUs, AV1, much faster than software encoders, though usually at a somewhat lower quality for the same bitrate. FFmpeg exposes it as encoders such as
h264_nvenc.NVIDIA GPUs in FFmpegGlossary (NVIDIA Video Encoder) and NVDECNVDEC: NVIDIA Video DecoderThe hardware video decoder of NVIDIA GPUs, a dedicated unit next to the CUDA cores. It decodes codecs such as H.264, HEVC, VP9 and, on newer GPUs, AV1, leaving the CPU and the CUDA cores free.NVIDIA GPUs in FFmpegGlossary (NVIDIA Video Decoder) from the host driver at run time. nvidia-smiworks inside the container. The harness uses it to find the GPUs; without it, the GPU scenarios are skipped.
The image sets NVIDIA_DRIVER_CAPABILITIES=compute,utility,video, so the
NVIDIA Container ToolkitNVIDIA Container ToolkitA set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container.NVIDIA docsGlossary also mounts the driver's video libraries without a
separate request. Without the video capability, the GPU
scenarios fail to load libnvcuvid.so.1 and are skipped.
Each GPU scenario first runs a one-worker probe. If the GPU cannot open an
NVENC or NVDEC session for it, the scenario is skipped with the error text as
its skip_reason.
Container Resource Limits
This benchmark sets its worker counts based on the resources available to the container, as reported by cgroup v2cgroup v2: control groups v2The Linux kernel feature that organizes processes into a hierarchy of groups, and limits and accounts for the CPU, memory and I/O of each group. Container runtimes such as Docker use it to enforce the resource limits of a container. It replaced the original cgroup v1 with a single, unified hierarchy.Linux kernel docsDocker resource constraintsGlossary (control groups v2) and CPU affinityCPU affinityThe set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread binding or CPU pinning.Linux man pageGlossary. The following Docker resource limits all lower the worker counts:
--cpus(cgroupcpu.max)--cpuset-cpus(CPU affinity)--memory(memory.max): each audio worker is budgeted 64 MiB and each video worker 320 MiB of available memory.--pids-limit(pids.max): each worker is budgetedFFMPEG_BENCH_PID_TASKS_PER_WORKERtasks (default: four), because the PIDPID: Process IdentifierThe number by which the operating system identifies a running process. On Linux, a PID is assigned when a process starts and can be reused after it exits, and commands such as kill use it to target a process.WikipediaGlossary (Process Identifier) limit counts threads, not only processes.
The output records the limits it found in host, including cgroup_cpu_max,
cgroup_cpuset_effective, and the cgroup_cpu_stat_before and
cgroup_cpu_stat_after snapshots. The _gb memory fields are in GiB.
The harness copies the test media into a work directory on /dev/shm when it
has at least 512 MiB free, and into the default temporary directory
otherwise. Docker's default /dev/shm is 64 MiB, so a plain docker run
uses /tmp.
Worker Counts
Each scenario is measured at several worker counts (parallel ffmpeg
processes), set in benchmark.py as follows:
- CPU scenarios: 1, P/2 (rounded down), and P, where P is the number of physical cores the container can use. Logical CPUs that are SMTSMT: Simultaneous MultithreadingA CPU technique that runs two (or more) hardware threads on one physical core, sharing its execution units, e.g. Intel Hyper-Threading. It raises throughput when a thread stalls, but the threads compete for the core. Cloud vendors usually sell each thread as a vCPU.WikipediaGlossary (Simultaneous Multithreading) siblings of the same core count once. If throughput at P is at least 1.08 times the throughput at P/2, one more point at 2·P is added, capped by the logical CPU count and the limits above.
- GPU scenarios: 1 and G, where G is the number of GPUs. The search then keeps doubling the worker count (2·G, 4·G, ...) as long as throughput does not drop, up to eight sessions per GPU by default. Any drop stops the search; the 1.08 ratio does not apply to GPUs.
A worker count with a failed worker stops the search for that scenario. On
servers with more than one NUMANUMA: Non-Uniform Memory AccessA memory design where each CPU socket or chiplet has its own local memory, which its cores reach faster than the memory of the others. Software that keeps its threads and their memory on the same NUMA node avoids the slower remote accesses.Linux kernel docsGlossary
(Non-Uniform Memory Access) node,
workers are spread across the nodes in proportion to their CPUs and bound to
them with numactl, if
binding works in the container. Otherwise, normal Linux scheduling applies.
For example, a server with 48 physical cores and 96 logical CPUs, with no container limits, measures the CPU scenarios at 1, 24, and 48 workers, and at 96 workers if 48 workers beat 24 by at least 8%.
The following settings change the worker counts:
FFMPEG_BENCH_WORKERSreplaces the automatic counts with an explicit, comma-separated list, capped at the maximum worker count, and disables the search.FFMPEG_BENCH_OVERSUBSCRIPTIONabove 1 adds one point at that multiple of P (or of G, for GPU scenarios).FFMPEG_BENCH_MAX_WORKERScaps the worker count.
Run Duration
Each scenario runs the following steps:
- A one-worker probe with 1 second of video or 5 seconds of audio checks that the scenario can run and estimates its speed. A scenario whose probe fails is skipped.
- At each worker count, a pilot run sizes the media for each worker so that
one repetition takes about
FFMPEG_BENCH_TARGET_SECONDS(default: 5s) of wall time. Faster servers therefore process more media in the same time. A successful pilot is not reported; a pilot that still fails after up to four attempts is reported as a failed repetition and ends the scenario. - By default, three repetitions are measured. While the CVCV: Coefficient of VariationThe standard deviation of a set of measurements divided by their mean, often given as a percentage. As it is scale-free, it compares the variability of measurements of different magnitudes, such as how much repeated runs of different benchmarks vary.WikipediaGlossary (Coefficient of Variation) of their rates is above 10%, more are added, up to five. A repetition with a failed worker ends the worker count.
Each repetition, including pilots, times out after
FFMPEG_BENCH_REPETITION_TIMEOUT_SECONDS (default: 30s), and workers that
run past it count as failed. The whole run stops after
FFMPEG_BENCH_TIMEOUT_SECONDS (default: 7,200s); scenarios that have not
started by then are reported as skipped with overall_timeout.
Total run time therefore depends on the number of scenarios that run and the
worker counts each one measures. A production run on an AWSAWS: Amazon Web ServicesThe cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning.AWSGlossary
(Amazon Web Services) m9g.16xlarge with 64 vCPUsvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary and no GPUs
measured every CPU and audio scenario at 1, 32, and 64 workers and took about
10 minutes
(recorded output).
Across 372 production runs of harness version 3.2.1, run time ranged from about 3 minutes on servers with 2 vCPUs to about 17 minutes on GPU servers, with a median of about 9.5 minutes.
Repetition Length
Each repetition targets 5 seconds of wall time by default. Because the pilot scales the media to that target, larger servers process more media in the same time instead of running longer.
The default comes from a study that compared targets of 2, 5, and 10 seconds on one developer workstation, a 24-core Intel Core i9-14900K desktop CPU without an NVIDIA GPU, so GPU scenarios were not part of it. Only the study's summary was kept; see Why 5 Seconds.
Shorter Test Runs
For a quick check that the image works, reduce the worker counts, repetitions, and media lengths:
docker run --rm \
-e FFMPEG_BENCH_WORKERS=1,2 \
-e FFMPEG_BENCH_REPETITIONS=1 \
-e FFMPEG_BENCH_TARGET_SECONDS=1 \
-e FFMPEG_BENCH_MIN_MEDIA_SECONDS=1 \
-e FFMPEG_BENCH_MAX_MEDIA_SECONDS=5 \
-e FFMPEG_BENCH_AUDIO_CALIBRATION_SECONDS=1 \
ghcr.io/sparecores/benchmark-ffmpeg:main
The results of such a run are not comparable with default runs.
Production Setup
The Inspector of Spare
Cores Navigator runs this image on every server type except those on the
inspector_rules.py
blocklist, with the settings in the following table. The last column shows what
a plain docker run does instead:
| Setting | Production | Plain docker run |
|---|---|---|
| Privileges and networking | Privileged, with host networking | Neither |
--shm-size | Not set | Not set |
| GPUs | Requested for the container, equivalent to --gpus all; the run is retried without GPUs if that fails | None, unless you pass --gpus all |
| Task timeout | 2 hours | 2 hours, the harness default |
| Minimum memory | Servers with less than 1 GiB of memory are skipped | Not enforced by the image |
This benchmark does not need privileged mode or host networking. With no
--shm-size set, /dev/shm keeps Docker's 64 MiB default and the test media is
copied to /tmp, in production and in a plain run.
We investigate failed production runs, manually or semi-automatically, and
then run them again. Until then, a server type's latest result in
sc-inspector-data can be a failed or partial run.
Environment Variables
The following variables configure this benchmark. See Results for the output fields.
| Variable | Meaning | Default |
|---|---|---|
FFMPEG_BENCH_TARGET_SECONDS | Target wall time per repetition in seconds. | 5 |
FFMPEG_BENCH_MIN_MEDIA_SECONDS | Minimum media length per worker in seconds. | 0.5 |
FFMPEG_BENCH_MAX_MEDIA_SECONDS | Maximum media length per worker in seconds. | 1800 |
FFMPEG_BENCH_VIDEO_CALIBRATION_SECONDS | Media length of the one-worker video probe in seconds. | 1 |
FFMPEG_BENCH_AUDIO_CALIBRATION_SECONDS | Media length of the one-worker audio probe in seconds. | 5 |
FFMPEG_BENCH_REPETITIONS | Measured repetitions per worker count (at least 1). | 3 |
FFMPEG_BENCH_MAX_REPETITIONS | Maximum repetitions per worker count when the rates vary. | 5 |
FFMPEG_BENCH_CV_THRESHOLD | Add repetitions while the coefficient of variation of the rates is above this value. | 0.10 |
FFMPEG_BENCH_OVERSUBSCRIPTION | Above 1, adds a worker count at this multiple of the physical cores (CPU) or GPUs (GPU). | 1 |
FFMPEG_BENCH_SCALE_CONTINUE_RATIO | CPU only: throughput gain from P/2 to P workers required to try 2·P workers. | 1.08 |
FFMPEG_BENCH_WORKERS | Comma-separated worker counts; replaces the automatic counts and disables the search. | Automatic |
FFMPEG_BENCH_MAX_WORKERS | Hard cap on the worker count. | Automatic |
FFMPEG_BENCH_GPU_ENCODE_SESSIONS_PER_GPU | Highest NVENC worker count per GPU the search can reach. | 8 |
FFMPEG_BENCH_GPU_DECODE_SESSIONS_PER_GPU | Highest NVDEC worker count per GPU the search can reach. | 8 |
FFMPEG_BENCH_PID_TASKS_PER_WORKER | Tasks budgeted per worker against the container's PID limit (at least 2). | 4 |
FFMPEG_BENCH_REPETITION_TIMEOUT_SECONDS | Timeout for each repetition or pilot in seconds. | 6 × target, at least 30 |
FFMPEG_BENCH_TIMEOUT_SECONDS | Deadline for the whole run in seconds. | 7200 |
FFMPEG_BENCH_AUDIO_SOURCE | Path of an alternate FLAC file. It is not checksum-verified. | Bundled sample |
FFMPEG_BENCH_VIDEO_SOURCE | Path of an alternate H.264H.264A lossy video codec standardized in 2003, also called AVC. It is the most widely supported video format, used in Blu-ray, broadcast, streaming, and video calls. Common encoders are x264 and hardware encoders such as NVENC.SpecificationGlossary MP4MP4: MPEG-4 Part 14A container file format that holds video, audio, and subtitle tracks, most often H.264 or HEVC video with AAC audio. It is the common format for video on the web and on phones.WikipediaGlossary file. It must match FFMPEG_BENCH_VIDEO_SHA256. | Bundled clip |
FFMPEG_BENCH_VIDEO_HEVC_SOURCE | Path of an alternate H.265H.265Another name for HEVC, from the number of its standard.SpecificationGlossary MP4 file. It must match FFMPEG_BENCH_VIDEO_HEVC_SHA256. | Bundled clip |
FFMPEG_BENCH_VIDEO_SHA256 | Expected SHA-256SHA-256: Secure Hash Algorithm 256A cryptographic hash function of the SHA-2 family that maps any data to a 256-bit digest. Even a one-bit change in the data gives a completely different digest, so it is used to verify that a downloaded file is intact, and in digital signatures and certificates.NIST FIPS 180-4Glossary (Secure Hash Algorithm 256) hash of the H.264 file; empty skips the check. | Pinned hash |
FFMPEG_BENCH_VIDEO_HEVC_SHA256 | Expected SHA-256 of the H.265 file; empty skips the check. | Pinned hash |
FFMPEG_BENCH_CDN_BASE_URL | Base URLURL: Uniform Resource LocatorThe address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.Glossary recorded as the test media source in the output; nothing is downloaded. | https://cdn.sparecores.net/sc-inspector/benchmark-ffmpeg |
FFMPEG, FFPROBE | Alternate ffmpeg and ffprobe binaries. | ffmpeg and ffprobe on PATH |