Usage
CPU Run
Run the image via Docker without options to measure the default models on the CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary:
docker run --rm ghcr.io/sparecores/benchmark-llm:main > results.jsonl
No options or environment variables are required. The :main tag has variants
for AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary and Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossary, and Docker pulls the variant for the host's architecture.
The Arm64 variant always uses the CPU build of llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary, and the AMD64 variant
uses it when the CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture) build cannot start,
for example without GPUs.
The --version option prints the commit hash of the llama.cpp build,
51f311e0, and exits:
docker run --rm ghcr.io/sparecores/benchmark-llm:main --version
GPU Run
To run on NVIDIA GPUs, the host needs the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary and the NVIDIA Container ToolkitNVIDIA Container ToolkitA set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container.NVIDIA docsGlossary. Pass the GPUs to the container:
docker run --rm --gpus all ghcr.io/sparecores/benchmark-llm:main > results.jsonl
Only the AMD64 variant of :main includes the CUDA build. At startup,
benchmark.py runs llama-cli --version from the CUDA
build. If it succeeds, this benchmark uses the CUDA build. If it fails, for
example on a non-NVIDIA GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary or with NVIDIA driver issues, this benchmark uses
the CPU build. This fallback runs inside the container, so it does not
replace the host requirements above. The choice is logged to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary, and each
output line records it in backends (CUDA or CPU).
With the CUDA build, this benchmark then finds how many layers of each model fit on the GPUs; see GPU Layers.
Model Downloads and Disk Space
This benchmark downloads models from Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary while it runs. It needs
network access to huggingface.co and to the Hugging Face CDNCDN: Content Delivery NetworkA network of distributed servers that cache content, such as web pages, images or video, and serve it from locations close to its users. It lowers latency for the users and offloads traffic from the origin server.WikipediaGlossary (Content
Delivery Network), such as us.aws.cdn.hf.co, unless the files are already in
--models-dir (default: /models). Downloads run one at a time in the
background while the benchmark measures models that have already finished
downloading. Each file is first saved as <file>.part and renamed when the
download completes.
The /models folder is a Docker volume, so a plain run downloads the models
again each time. To reuse them across runs, mount a host directory on it:
docker run --rm -v "$PWD/models:/models" ghcr.io/sparecores/benchmark-llm:main > results.jsonl
Together, the six default models take ~58 GB, the largest ~42 GB. Before each
download, this benchmark reads the file size from Hugging Face and skips the
model if the free space is less than the file size plus 10%. The check reads
the free space of the container's root filesystem (/), not of
--models-dir, so with a separately mounted models directory, it does not
reflect that directory's free space. Check the free space of the mounted
directory yourself.
The --download-timeout option (default: 900s) limits the wait for each
model. Once at least one download has finished, this benchmark also estimates
from the average download speed whether the next model can finish downloading
within that time. If a model would not finish, or has not finished when the
wait ends, this benchmark stops the downloads and exits with code 0, skipping
that model and all larger ones. The check runs only after all earlier models
have been measured, so their results are complete.
On SIGINTSIGINTThe interrupt signal of POSIX systems, which a terminal sends to the running process when the user presses Ctrl+C. A process can catch it to clean up before it exits; by convention, a process that exits because of a signal returns 128 plus the signal number, 130 for SIGINT.signal(7) man pageGlossary or SIGTERMSIGTERMThe termination signal of POSIX systems, which asks a process to exit cleanly. It is the default signal of the kill command, and docker stop sends it to a container's main process first, then SIGKILL, which cannot be caught, if the process has not exited after a grace period. By convention, a process that exits because of it returns 143 (128 plus 15).signal(7) man pageGlossary, this benchmark deletes the incomplete .part files
and exits with code 128 plus the signal number.
Run Duration
Run time depends on the download speed, on the server's speed, and on how many models and tokentokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary counts fit in their timeouts; see Timeouts and Early Stops. Sample production runs on AWSAWS: Amazon Web ServicesThe cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning.AWSGlossary (Amazon Web Services) took the following times. Each server type links to its LLMLLM: Large Language ModelA neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code.WikipediaGlossary scores in Spare Cores Navigator, and each run to its production output:
c7g.large: 18 min, stopped at the 7B model (run)c9g.48xlarge: 26 min, stopped at the 70B model's download (run)m7i.2xlarge: 31 min (run)g4dn.xlargewith an NVIDIA GPU: 32 min (run)
To give slower servers more time, raise --benchmark-timeout-scale, which
multiplies every measurement's timeout without changing the llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary
options. Such a run can then measure models and token counts that a default
run skips.
Production Setup
The Navigator's Inspector
(sc-inspector) runs this image
on every server type except those on the
inspector_rules.py
blocklist, with the following settings:
| Setting | Production | Plain docker run |
|---|---|---|
| Privileges | Privileged | Not privileged |
| Networking | Host networking | Docker's default bridge network |
| GPUs | All GPUs, as with --gpus all. If that run fails or times out, it is retried without GPUs. | None, unless --gpus is passed |
| Overall timeout | 1.5 hours | None, only per-measurement and download timeouts |
| Memory minimum | Skips servers with less than 1 GiB | None |
| Options | None, so the default models and timeouts | As passed |
To reproduce the container settings above, run the following command on a
server with GPUs. Omit --gpus all on a server without GPUs:
docker run --rm --privileged --network host --gpus all ghcr.io/sparecores/benchmark-llm:main > results.jsonl
We investigate failed production runs, manually or semi-automatically, and
then run them again. Until then, a server type's latest result in
sc-inspector-data can be a failed or partial run.
Options
Pass the following options after the image name. This benchmark has no environment variables of its own. See Results for the output fields.
| Option | Meaning | Default |
|---|---|---|
--model-urls | URLsURL: Uniform Resource LocatorThe address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.Glossary of GGUFGGUF: GGML Universal FileA binary file format for LLM weights and metadata, introduced in 2023 by the ggml project behind llama.cpp. One file holds the model's tensors, often quantized, together with its tokenizer and settings, so a model can be loaded quickly with memory mapping and without other files. Hugging Face hosts many models in this format.GGUF specificationHugging Face docsGlossary (GGMLGGML: Georgi Gerganov Machine LearningAn open-source tensor library for machine learning, written in C and C++ without external dependencies. It runs on CPUs of many architectures, with SIMD-optimized code, and on GPUs and other accelerators through backends, and it supports quantized weights of 2 to 8 bits. llama.cpp and the GGUF file format are built on it.ggml on GitHubGlossary Universal File) model files to download and measure, in order, separated by spaces after one flag. Order them from smallest to largest: a model that fails its first measurement stops the run. | The six default models |
--models-dir | Directory to download and cache the models in. | /models |
--benchmark-timeout-scale | Integer that multiplies every measurement's timeout. | 1 |
--download-timeout | Maximum wait for each model download in seconds. This benchmark also gives up a model when its remaining download, at the average speed of the earlier downloads, would take longer. | 900 |
--version | Print the llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary commit hash and exit; unknown if it cannot be read. | Off |
To measure other models, pass their URLs after one --model-urls flag. For
example, the following command measures only the two smallest default models:
docker run --rm ghcr.io/sparecores/benchmark-llm:main --model-urls \
https://huggingface.co/QuantFactory/SmolLM-135M-GGUF/resolve/b35f229108e4c5a4deeccccf9bf75cd46006fa7b/SmolLM-135M.Q4_K_M.gguf \
https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat-GGUF/resolve/cfab082d2fef4a8736ef384dc764c2fb6887f387/qwen1_5-0_5b-chat-q4_k_m.gguf \
> results.jsonl