Skip to main content

Usage

CPU Run​

Run the image via Docker without options to measure the default models on the CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary:

docker run --rm ghcr.io/sparecores/benchmark-llm:main > results.jsonl

No options or environment variables are required. The :main tag has variants for AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary and Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossary, and Docker pulls the variant for the host's architecture. The Arm64 variant always uses the CPU build of llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary, and the AMD64 variant uses it when the CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture) build cannot start, for example without GPUs.

The --version option prints the commit hash of the llama.cpp build, 51f311e0, and exits:

docker run --rm ghcr.io/sparecores/benchmark-llm:main --version

GPU Run​

To run on NVIDIA GPUs, the host needs the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary and the NVIDIA Container ToolkitNVIDIA Container ToolkitA set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container.NVIDIA docsGlossary. Pass the GPUs to the container:

docker run --rm --gpus all ghcr.io/sparecores/benchmark-llm:main > results.jsonl

Only the AMD64 variant of :main includes the CUDA build. At startup, benchmark.py runs llama-cli --version from the CUDA build. If it succeeds, this benchmark uses the CUDA build. If it fails, for example on a non-NVIDIA GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary or with NVIDIA driver issues, this benchmark uses the CPU build. This fallback runs inside the container, so it does not replace the host requirements above. The choice is logged to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary, and each output line records it in backends (CUDA or CPU).

With the CUDA build, this benchmark then finds how many layers of each model fit on the GPUs; see GPU Layers.

Model Downloads and Disk Space​

This benchmark downloads models from Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary while it runs. It needs network access to huggingface.co and to the Hugging Face CDNCDN: Content Delivery NetworkA network of distributed servers that cache content, such as web pages, images or video, and serve it from locations close to its users. It lowers latency for the users and offloads traffic from the origin server.WikipediaGlossary (Content Delivery Network), such as us.aws.cdn.hf.co, unless the files are already in --models-dir (default: /models). Downloads run one at a time in the background while the benchmark measures models that have already finished downloading. Each file is first saved as <file>.part and renamed when the download completes.

The /models folder is a Docker volume, so a plain run downloads the models again each time. To reuse them across runs, mount a host directory on it:

docker run --rm -v "$PWD/models:/models" ghcr.io/sparecores/benchmark-llm:main > results.jsonl

Together, the six default models take ~58 GB, the largest ~42 GB. Before each download, this benchmark reads the file size from Hugging Face and skips the model if the free space is less than the file size plus 10%. The check reads the free space of the container's root filesystem (/), not of --models-dir, so with a separately mounted models directory, it does not reflect that directory's free space. Check the free space of the mounted directory yourself.

The --download-timeout option (default: 900s) limits the wait for each model. Once at least one download has finished, this benchmark also estimates from the average download speed whether the next model can finish downloading within that time. If a model would not finish, or has not finished when the wait ends, this benchmark stops the downloads and exits with code 0, skipping that model and all larger ones. The check runs only after all earlier models have been measured, so their results are complete.

On SIGINTSIGINTThe interrupt signal of POSIX systems, which a terminal sends to the running process when the user presses Ctrl+C. A process can catch it to clean up before it exits; by convention, a process that exits because of a signal returns 128 plus the signal number, 130 for SIGINT.signal(7) man pageGlossary or SIGTERMSIGTERMThe termination signal of POSIX systems, which asks a process to exit cleanly. It is the default signal of the kill command, and docker stop sends it to a container's main process first, then SIGKILL, which cannot be caught, if the process has not exited after a grace period. By convention, a process that exits because of it returns 143 (128 plus 15).signal(7) man pageGlossary, this benchmark deletes the incomplete .part files and exits with code 128 plus the signal number.

Run Duration​

Run time depends on the download speed, on the server's speed, and on how many models and tokentokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary counts fit in their timeouts; see Timeouts and Early Stops. Sample production runs on AWSAWS: Amazon Web ServicesThe cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning.AWSGlossary (Amazon Web Services) took the following times. Each server type links to its LLMLLM: Large Language ModelA neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code.WikipediaGlossary scores in Spare Cores Navigator, and each run to its production output:

To give slower servers more time, raise --benchmark-timeout-scale, which multiplies every measurement's timeout without changing the llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary options. Such a run can then measure models and token counts that a default run skips.

Production Setup​

The Navigator's Inspector (sc-inspector) runs this image on every server type except those on the inspector_rules.py blocklist, with the following settings:

SettingProductionPlain docker run
PrivilegesPrivilegedNot privileged
NetworkingHost networkingDocker's default bridge network
GPUsAll GPUs, as with --gpus all. If that run fails or times out, it is retried without GPUs.None, unless --gpus is passed
Overall timeout1.5 hoursNone, only per-measurement and download timeouts
Memory minimumSkips servers with less than 1 GiBNone
OptionsNone, so the default models and timeoutsAs passed

To reproduce the container settings above, run the following command on a server with GPUs. Omit --gpus all on a server without GPUs:

docker run --rm --privileged --network host --gpus all ghcr.io/sparecores/benchmark-llm:main > results.jsonl

We investigate failed production runs, manually or semi-automatically, and then run them again. Until then, a server type's latest result in sc-inspector-data can be a failed or partial run.

Options​

Pass the following options after the image name. This benchmark has no environment variables of its own. See Results for the output fields.

OptionMeaningDefault
--model-urlsURLsURL: Uniform Resource LocatorThe address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.Glossary of GGUFGGUF: GGML Universal FileA binary file format for LLM weights and metadata, introduced in 2023 by the ggml project behind llama.cpp. One file holds the model's tensors, often quantized, together with its tokenizer and settings, so a model can be loaded quickly with memory mapping and without other files. Hugging Face hosts many models in this format.GGUF specificationHugging Face docsGlossary (GGMLGGML: Georgi Gerganov Machine LearningAn open-source tensor library for machine learning, written in C and C++ without external dependencies. It runs on CPUs of many architectures, with SIMD-optimized code, and on GPUs and other accelerators through backends, and it supports quantized weights of 2 to 8 bits. llama.cpp and the GGUF file format are built on it.ggml on GitHubGlossary Universal File) model files to download and measure, in order, separated by spaces after one flag. Order them from smallest to largest: a model that fails its first measurement stops the run.The six default models
--models-dirDirectory to download and cache the models in./models
--benchmark-timeout-scaleInteger that multiplies every measurement's timeout.1
--download-timeoutMaximum wait for each model download in seconds. This benchmark also gives up a model when its remaining download, at the average speed of the earlier downloads, would take longer.900
--versionPrint the llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary commit hash and exit; unknown if it cannot be read.Off

To measure other models, pass their URLs after one --model-urls flag. For example, the following command measures only the two smallest default models:

docker run --rm ghcr.io/sparecores/benchmark-llm:main --model-urls \
https://huggingface.co/QuantFactory/SmolLM-135M-GGUF/resolve/b35f229108e4c5a4deeccccf9bf75cd46006fa7b/SmolLM-135M.Q4_K_M.gguf \
https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat-GGUF/resolve/cfab082d2fef4a8736ef384dc764c2fb6887f387/qwen1_5-0_5b-chat-q4_k_m.gguf \
> results.jsonl