Skip to main content

llama.cpp

This benchmark measures single-stream LLMLLM: Large Language ModelA neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code.WikipediaGlossary inference with llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary from llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary. It downloads six 4-bit quantizedquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary models, ranging from 135 million to 70 billion parameters, and tests them from smallest to largest. It uses NVIDIA GPUs when the CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture) build of llama.cpp starts successfully; otherwise, it runs on the CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary.

This benchmark reports speed in tokenstokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary per second for two tasks, prompt processingprompt processingThe first phase of LLM inference, also called prefill, in which the model reads all tokens of the input prompt and stores their intermediate results in its key-value cache. The prompt's tokens are processed together in batches, so this phase is usually limited by compute and runs much faster per token than text generation. Its speed decides how long a long input takes before the first output token appears.Glossary and text generationtext generationThe second phase of LLM inference, also called decoding, in which the model produces output tokens one at a time, each depending on the tokens before it. Every step reads all model weights and the key-value cache, so this phase is usually limited by memory bandwidth rather than compute, and runs much slower per token than prompt processing.Glossary, each at several token counts. It prints one JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary line per model, task, and token count, with the values of five repetitions and their average.

Purpose​

Spare Cores Navigator publishes empirical performance measurements for more than 5,000 cloud server types, including LLM inference speedLLM inference speedHow fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generation, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.Glossary from this benchmark: one score in tokens per second for each server type, model, task, and token count. In the Navigator's Web UI, you can order the server list by one of these scores, and each server's details page charts all of them, for example for m7i.2xlarge.

We needed one method that could run on every server type, from hosts with less than 1 GB of RAMRAM: Random-Access MemoryThe main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off.WikipediaGlossary to hosts with terabytes of memory, hundreds of CPU cores, and several GPUs. We chose llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary because it runs inference on both CPUs and GPUs with minimal tuning. The project also publishes container images for AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary and Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossary, a CUDA build, and the configurable llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary tool.

For a single user, llama.cpp is often the fastest self-hosted option, so this benchmark measures one request stream at a time. For shared serving with many concurrent requests, vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary is preferred; the vLLM benchmarks cover that case, but the Navigator does not show their results yet.

Limitations​

These results compare single-stream LLM inference speed across server types. Note the following limitations:

  • Concurrent Serving: This benchmark runs one llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary process at a time and sends no concurrent requests, so it does not measure serving throughput under load.
  • Output Quality: This benchmark measures speed in tokens per second, not the quality or accuracy of the model output.
  • Fixed Models and Build: Every run uses the same six models, 4-bit quantizationquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary, and llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary build b4753. Other models, quantization levels, and builds are not covered.
  • CPU or GPU Results: On a GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary server, a score can come from the GPU or from the CPU fallback, and the Navigator does not mark which. Only NVIDIA GPUs are used, and only on AMD64. See also GPU Coverage.
  • Missing Results: On slow servers, timeouts skip larger token counts and models, and the run still exits with code 0. The Navigator shows "no data" for the skipped results.
  • Downloads and Disk Space: The models are downloaded during the run, so a slow download can skip the largest models, even on a fast server. A model is also skipped if the disk has too little free space for it.

Usage​

Run the image via Docker. Without GPU access, it runs the CPU build of llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary:

docker run --rm ghcr.io/sparecores/benchmark-llm:main > results.jsonl

The image downloads the models from Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary while it runs, so it needs network access and up to ~58 GB of free disk space for all six models. It caches them in /models, a Docker volume. See Model Downloads and Disk Space to reuse downloaded models.

To run on NVIDIA GPUs, the host needs the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary and the NVIDIA Container ToolkitNVIDIA Container ToolkitA set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container.NVIDIA docsGlossary. Pass the GPUs to the container. Only the AMD64 variant of :main includes the CUDA build:

docker run --rm --gpus all ghcr.io/sparecores/benchmark-llm:main > results.jsonl

If the CUDA build cannot start inside the container, for example on a non-NVIDIA GPU or with NVIDIA driver issues, this benchmark falls back to the CPU build. The fallback does not replace the host requirements above. See GPU Run for how it picks the build and the GPU layers.

Production runs the container privileged, with host networking, and with the GPUs passed on GPU servers. A plain docker run uses neither privileged mode nor host networking. See Production Setup before you compare your results with the Navigator's.

Sample production runs took about 18 to 32 minutes. Run time depends on the download speed, on the server's speed, and on how many models and token counts fit in their timeouts. See Run Duration for the timeouts.

Settings​

Pass options after the image name. The following options are the most common; see the full option reference for all options and defaults:

  • --model-urls replaces the default model list with URLsURL: Uniform Resource LocatorThe address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.Glossary of GGUFGGUF: GGML Universal FileA binary file format for LLM weights and metadata, introduced in 2023 by the ggml project behind llama.cpp. One file holds the model's tensors, often quantized, together with its tokenizer and settings, so a model can be loaded quickly with memory mapping and without other files. Hugging Face hosts many models in this format.GGUF specificationHugging Face docsGlossary (GGMLGGML: Georgi Gerganov Machine LearningAn open-source tensor library for machine learning, written in C and C++ without external dependencies. It runs on CPUs of many architectures, with SIMD-optimized code, and on GPUs and other accelerators through backends, and it supports quantized weights of 2 to 8 bits. llama.cpp and the GGUF file format are built on it.ggml on GitHubGlossary Universal File) model files, separated by spaces and ordered from smallest to largest.
  • --models-dir sets the download and cache directory (default: /models).
  • --benchmark-timeout-scale multiplies every per-scenario timeout by an integer (default: 1).
  • --download-timeout sets the maximum wait for each model download (default: 900s).
  • --version prints the commit hash of llama.cpp build b4753, 51f311e0, and exits.

Results​

The process prints JSON LinesJSON LinesA text format that stores one JSON value, usually an object, per line, so a file can be written and read one record at a time. It suits logs and streamed results, and line-based tools such as grep can process it. Files in this format usually end in .jsonl.JSON LinesGlossary to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary: one line per model, task, and token count, written as each measurement finishes. Logs go to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary. The main fields of each line are the following:

  • samples_ts lists the speeds of the five repetitions in tokens per second, avg_ts their average, and stddev_ts their standard deviation.
  • n_prompt and n_gen identify the task: a prompt processing line has n_gen: 0 and n_prompt tokens, and a text generation line has n_prompt: 0 and n_gen tokens.
  • model_filename, model_size (in bytes), and model_n_params identify the model.
  • backends is CUDA or CPU, and n_gpu_layers is the number of model layers offloaded to the GPUs. The gpu_info, cpu_info, and n_threads fields describe the hardware used.
  • samples_ns gives the time of each repetition in nanoseconds, and avg_ns their average.

For example, on an AWSAWS: Amazon Web ServicesThe cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning.AWSGlossary (Amazon Web Services) m7i.2xlarge, the 7B model processed a 512-token prompt at 53.75 tokens/s and generated 128 tokens at 7.71 tokens/s (output).

The process exits with code 0 even when it skipped models or token counts, so check which lines are present, and the reasons logged to stderr. On SIGINTSIGINTThe interrupt signal of POSIX systems, which a terminal sends to the running process when the user presses Ctrl+C. A process can catch it to clean up before it exits; by convention, a process that exits because of a signal returns 128 plus the signal number, 130 for SIGINT.signal(7) man pageGlossary or SIGTERMSIGTERMThe termination signal of POSIX systems, which asks a process to exit cleanly. It is the default signal of the kill command, and docker stop sends it to a container's main process first, then SIGKILL, which cannot be caught, if the process has not exited after a grace period. By convention, a process that exits because of it returns 143 (128 plus 15).signal(7) man pageGlossary, it exits with code 128 plus the signal number. Resource usage is recorded with resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary, which streams it to the Spare Cores Sentinel platform for the team's own monitoring. It is not published.

Workloads​

Each model runs the following two tasks, in this order:

  • Prompt Processing: processes a prompt of 16 to 16,384 tokens without generating any (-p <n> -n 0).
  • Text Generation: generates 16 to 4,096 tokens without a prompt (-n <n> -p 0).

Each token count runs as a separate llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary process with its own timeout. See Models for the six default models, and Timeouts and Early Stops for how slow servers skip larger token counts and models.

Design History​

We selected widely used LLMs ranging from 135 million to 70 billion parameters. Their 4-bit quantizedquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary versions use less memory and load faster. We chose llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary because it runs on CPUs and GPUs across the full range of server types. We pinned build b4753, the latest build that worked in our initial tests.

We do not plan to update the pinned build: comparable results would require rerunning the benchmark on every server type already measured, potentially costing tens of thousands of US dollars. We would reconsider if a newer build materially changed the performance differences between server types, not just their overall speed.

Long prompts can take hours to process on slow servers. We therefore expect speed to increase with token count: 1 token/s may be acceptable for a short response, but not for batch processing long inputs. In experiments across several cloud server types, speed usually increased with token count up to 1,024 tokens, so early stops begin at that size.

In production, speed and download time limit how many models a server completes. On an AWS c7g.large, the 7B model timed out on its first scenario (log), and on a c9g.48xlarge, this benchmark gave up the 70B model's download at 25.7 MB/s (log). See the LLM inference speed article for the methodology and results in more detail.

References​

See the full list of references used in this documentation. To change or rebuild this image, see the contributor guide.