Limitations
Disclaimer: What This Benchmark Does Not Deliver
Concurrent Serving
This benchmark runs one llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary process at a time, and the
harness sends no concurrent requests. The results show the
speed a single user gets from a model, not the throughput of a server that
answers many requests at once, such as a shared LLMLLM: Large Language ModelA neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code.WikipediaGlossary service.
For a single user, llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary is often the fastest self-hosted option, but it
does not scale as well to many concurrent requests, where vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary is preferred.
The vLLM benchmarks measure that case;
Spare Cores Navigator does not show
their results yet.
Output Quality
This benchmark measures speed in tokenstokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary per second. It does not check the quality or accuracy of the model output.
Scope Decisions
Fixed Models and Build
Every run uses the same six models, all 4-bit quantizedquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary, and the same llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary
build, b4753 (commit 51f311e0). The models are a diverse set of widely used
LLMs, from 135 million to 70 billion parameters, and their quantized versions
need much less memory and load faster. See Models for
the list.
Other models, quantization levels, and llama.cpp builds are not covered. We do
not plan to update the build or the models, because comparable results would
need a rerun on every server type already measured. A newer llama.cpp build may
be faster in general; the published results show the differences between server
types at build b4753.
The --model-urls option replaces the model list for your own runs. The
results of such a run are not comparable with the Navigator's.
CPU or GPU Results
On AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary, this benchmark first tries the CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device
Architecture) build of llama.cpp, and falls back to the CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary build if it cannot
start, for example on a non-NVIDIA GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary or with NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary issues. Production
also retries a failed GPU run without GPUs. See
GPU Run and
Production Setup.
The Navigator publishes the score measured on a server type, without marking
whether it came from the GPU or the CPU. Each output line records it in
backends (CUDA or CPU), but this field is not published.
GPU Coverage
Only NVIDIA GPUs are used, through CUDA. The Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossary image has no CUDA build, so
it always runs on the CPU.
When a model does not fit in GPU memory, this benchmark offloads fewer of its
layers to the GPUs, and the remaining layers stay on the CPU. Such a score
depends on both, and n_gpu_layers in the output shows how many layers were
offloaded.
See GPU Layers.
Design Constraints
Missing Results
A slow server cannot spend hours on long prompts or large models, so every measurement has a timeout, and the following rules skip the rest of a run:
- If the first, 16-token measurement of either task fails or times out, this benchmark stops and skips all larger models.
- If a later token count fails or times out, the remaining token counts of that task are skipped for that model.
- If a measurement of 1,024 or more tokens takes too long to meet the next token count's expected speed, the remaining token counts of that task are skipped.
- If a model would not finish downloading within
--download-timeout, the run stops, so that model and all larger ones are skipped. - If the disk has too little free space for a model, only that model is skipped, and the run continues with the next one. The next default models are larger, so they are skipped too.
All of these paths exit with code 0, so a run with missing results looks like a successful run. See Timeouts and Early Stops for the timeouts.
The Navigator shows "no data" for a missing score, because the cause is not
recorded: the server can be too slow, or the run can have failed or not tried.
The Navigator's LLM workload profile includes a penalty when the smaller models
have scores but the larger ones are missing, because then the larger models
could not load or finish in time. For example, the
AWS (Amazon Web Services) c7g.large page
has scores for the smaller models only.
Downloads and Disk Space
The models are downloaded from Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary during the run, not built into the
image, so download speed affects which models a run reaches. In production, an
AWS c9g.48xlarge downloaded at 25.7 MB/s and gave up the 70B model, whose
file is ~42 GB
(log).
Its LLM scores
stop at the 14B model.
Together, the six default models take ~58 GB of disk space. This benchmark
checks the free space of the container's root filesystem, not of --models-dir;
see Model Downloads and Disk Space.