Workloads
How a Run Works
Each run of the harness goes through the following steps:
- It logs the CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary model and flags to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary, and picks the CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute
Unified Device Architecture) or the CPU build of llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool,
llama-bench.llama.cpp on GitHubGlossary; see GPU Run. - It starts downloading all models in one background process, at low
priority (
nice 19niceA Unix command that runs a program with a changed scheduling priority, from -20 (highest) to 19 (lowest). Raising a process's priority above the default needs theCAP_SYS_NICEcapability, which root has.man pageGlossary), one after another in list order. Model files already in--models-dirare reused. - For each model, in list order, it waits for the model's download, finds the number of GPU layers (CUDA build only), and then runs prompt processing and text generation, each over increasing tokentokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary counts.
Each measurement of one model, task, and token count runs as a separate
llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary process, one at a time, so that it can have its own timeout. A
timeout or failure skips the rest of the task or the run; see
Timeouts and Early Stops.
Models
The default models are the following GGUFGGUF: GGML Universal FileA binary file format for LLM weights and metadata, introduced in 2023 by the ggml project behind llama.cpp. One file holds the model's tensors, often quantized, together with its tokenizer and settings, so a model can be loaded quickly with memory mapping and without other files. Hugging Face hosts many models in this format.GGUF specificationHugging Face docsGlossary (GGMLGGML: Georgi Gerganov Machine LearningAn open-source tensor library for machine learning, written in C and C++ without external dependencies. It runs on CPUs of many architectures, with SIMD-optimized code, and on GPUs and other accelerators through backends, and it supports quantized weights of 2 to 8 bits. llama.cpp and the GGUF file format are built on it.ggml on GitHubGlossary Universal File) files, from smallest to largest. The file sizes are approximate:
| Model | Parameters | Quantization | File size | Download |
|---|---|---|---|---|
| SmolLM | 135M | Q4_K_MQ4_K_MA llama.cpp quantization type of the k-quant family. It stores most weights with about 4 bits each, in blocks that share scale factors, and keeps some tensors at higher precision. The M (medium) variant is a common compromise between file size and output quality.Hugging Face GGUF docsGlossary | ~100 MB | GGUF |
| Qwen1.5-Chat | 0.5B | Q4_K_M | ~400 MB | GGUF |
| Gemma | 2B | Q4_K_M | ~1.5 GB | GGUF |
| LLaMA | 7B | Q4_K_M | ~4 GB | GGUF |
| Phi-4 | 14B | 4-bit (q4) | ~9 GB | GGUF |
| Llama-3.3-Instruct | 70B | Q4_K_M | ~42 GB | GGUF |
Each URLURL: Uniform Resource LocatorThe address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.Glossary is pinned to a commit of its Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary repository, so every run downloads the same files. The order matters: a model that fails its first measurement stops the run and skips all larger models.
Prompt Processing
Prompt processing measures how fast the model reads its input. Each
measurement processes a prompt of 16, 128, 512, 1,024, 4,096, or 16,384 tokens
without generating any (llama-bench -p <n> -n 0llama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary). Its output lines have
n_gen: 0, and n_prompt is the token count.
Text Generation
Text generation measures how fast the model writes its output, one token after
another. Each measurement generates 16, 128, 512, 1,024, or 4,096 tokens without
a prompt (llama-bench -n <n> -p 0llama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary). Its output lines have n_prompt: 0, and
n_gen is the token count.
llama-bench Settings
Every llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary measurement uses the following settings, set in
benchmark.py:
| Setting | Option | Value |
|---|---|---|
| Threads | -t | The number of physical CPU cores, as psutilpsutilA cross-platform Python library for retrieving information on running processes and system utilization, such as CPU, memory, disks, and network. It can count both logical CPUs and physical cores.psutil docsGlossary counts them; recorded as n_threads |
| Split mode | -sm-smThe llama.cpp option that sets the split mode: how a model is distributed across several GPUs. The layer mode assigns whole layers to each GPU, row splits the weight tensors of each layer across the GPUs, and none uses a single GPU.llama-bench READMEGlossary | layer |
| Flash attention | -fa | 1 (on) |
| Logical batch size | -b-bThe llama.cpp option that sets the logical batch size: the maximum number of prompt tokens passed to the model in one call. Each logical batch is split into physical batches of the size set by -ub, so a larger logical batch can process long prompts in fewer calls, at the cost of more memory.llama-bench READMEGlossary | 2,048 |
| Physical batch size | -ub-ubThe llama.cpp option that sets the physical batch size: the number of tokens computed at once within a logical batch (-b). It trades memory use against the speed of prompt processing.llama-bench READMEGlossary | 512 |
| Repetitions | -r | Not set, so llama-bench's default applies; production output has five repetitions per line |
| Output | -o | jsonl (JSON LinesJSON LinesA text format that stores one JSON value, usually an object, per line, so a file can be written and read one record at a time. It suits logs and streamed results, and line-based tools such as grep can process it. Files in this format usually end in .jsonl.JSON LinesGlossary), printed to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary as is |
GPU Layers
With the CUDA build, this benchmark sets how many model layers to offload to
the GPUs (-ngl-nglThe llama.cpp option that sets how many layers of a model are offloaded to the GPUs. The remaining layers run on the CPU, so a model that does not fit in GPU memory can still use the GPU in part, and a value higher than the model's number of layers, such as 999, offloads all of them.llama-bench READMEGlossary) before it measures each model. It tries 999, 40, 24, and 12,
in this order. For each value, it runs llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary with one repetition, one
thread, and a fixed timeout of 120 seconds, and uses the first value whose run
succeeds. If none succeeds, or with the CPU build, it uses 0. The value is
recorded as n_gpu_layers.
Timeouts and Early Stops
Each measurement's timeout in seconds depends on the model size, the token count, and the expected speed for that task and token count:
timeout = round((model_GiB / 0.25 + 1 + 5 × tokens / expected_speed) × scale)
The model_GiB variable is the model file's size in GiB. The formula allows 4
seconds per GiB to load the model, 1 second of overhead, and five repetitions
at the expected speed. The scale variable is the value of
--benchmark-timeout-scale (default: 1). The expected speeds rise with the
token count:
| Tokens | Prompt processing (tokens/s) | Text generation (tokens/s) |
|---|---|---|
| 16 | 2 | 1 |
| 128 | 10 | 5 |
| 512 | 25 | 25 |
| 1,024 | 50 | 50 |
| 4,096 | 250 | 250 |
| 16,384 | 1,000 | Not run |
For example, text generation of 16 tokens with a model of 4 GiB gets
4 / 0.25 + 1 + 5 × 16 / 1 = 97 seconds, and at 512 tokens 119 seconds. A
server that generates 15 tokens/s needs ~171 seconds for the five
repetitions at 512 tokens, so its measurement times out and the larger token
counts of that task are skipped. The load allowance adds some slack: a server
slightly below the expected speed can still finish in time.
After a successful measurement of 1,024 or more tokens that is not the last of its task, this benchmark also estimates whether the next token count can meet its expected speed. If the measurement took longer than the following limit, the remaining token counts of that task are skipped:
limit = timeout × expected_speed / next_expected_speed × 1.1
The timeout and expected_speed variables are those of the measurement that
just ran, and next_expected_speed is the expected speed of the next token
count. For prompt processing at 1,024 tokens, that limit is 22% of the timeout
(50 / 250 × 1.1).
When a measurement times out or fails, the following happens:
- On the first, 16-token measurement of either task, this benchmark stops and skips all larger models.
- On a later token count, the remaining token counts of that task are skipped for that model.
To allow slower servers more time, raise --benchmark-timeout-scale. See
Missing Results for the other reasons a
run can stop early and how
Spare Cores Navigator shows them.