Skip to main content

Workloads

How a Run Works​

Each run of the harness goes through the following steps:

  1. It logs the CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary model and flags to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary, and picks the CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture) or the CPU build of llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary; see GPU Run.
  2. It starts downloading all models in one background process, at low priority (nice 19niceA Unix command that runs a program with a changed scheduling priority, from -20 (highest) to 19 (lowest). Raising a process's priority above the default needs the CAP_SYS_NICE capability, which root has.man pageGlossary), one after another in list order. Model files already in --models-dir are reused.
  3. For each model, in list order, it waits for the model's download, finds the number of GPU layers (CUDA build only), and then runs prompt processing and text generation, each over increasing tokentokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary counts.

Each measurement of one model, task, and token count runs as a separate llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary process, one at a time, so that it can have its own timeout. A timeout or failure skips the rest of the task or the run; see Timeouts and Early Stops.

Models​

The default models are the following GGUFGGUF: GGML Universal FileA binary file format for LLM weights and metadata, introduced in 2023 by the ggml project behind llama.cpp. One file holds the model's tensors, often quantized, together with its tokenizer and settings, so a model can be loaded quickly with memory mapping and without other files. Hugging Face hosts many models in this format.GGUF specificationHugging Face docsGlossary (GGMLGGML: Georgi Gerganov Machine LearningAn open-source tensor library for machine learning, written in C and C++ without external dependencies. It runs on CPUs of many architectures, with SIMD-optimized code, and on GPUs and other accelerators through backends, and it supports quantized weights of 2 to 8 bits. llama.cpp and the GGUF file format are built on it.ggml on GitHubGlossary Universal File) files, from smallest to largest. The file sizes are approximate:

ModelParametersQuantizationFile sizeDownload
SmolLM135MQ4_K_MQ4_K_MA llama.cpp quantization type of the k-quant family. It stores most weights with about 4 bits each, in blocks that share scale factors, and keeps some tensors at higher precision. The M (medium) variant is a common compromise between file size and output quality.Hugging Face GGUF docsGlossary~100 MBGGUF
Qwen1.5-Chat0.5BQ4_K_M~400 MBGGUF
Gemma2BQ4_K_M~1.5 GBGGUF
LLaMA7BQ4_K_M~4 GBGGUF
Phi-414B4-bit (q4)~9 GBGGUF
Llama-3.3-Instruct70BQ4_K_M~42 GBGGUF

Each URLURL: Uniform Resource LocatorThe address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.Glossary is pinned to a commit of its Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary repository, so every run downloads the same files. The order matters: a model that fails its first measurement stops the run and skips all larger models.

Prompt Processing​

Prompt processing measures how fast the model reads its input. Each measurement processes a prompt of 16, 128, 512, 1,024, 4,096, or 16,384 tokens without generating any (llama-bench -p <n> -n 0llama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary). Its output lines have n_gen: 0, and n_prompt is the token count.

Text Generation​

Text generation measures how fast the model writes its output, one token after another. Each measurement generates 16, 128, 512, 1,024, or 4,096 tokens without a prompt (llama-bench -n <n> -p 0llama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary). Its output lines have n_prompt: 0, and n_gen is the token count.

llama-bench Settings​

Every llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary measurement uses the following settings, set in benchmark.py:

SettingOptionValue
Threads-tThe number of physical CPU cores, as psutilpsutilA cross-platform Python library for retrieving information on running processes and system utilization, such as CPU, memory, disks, and network. It can count both logical CPUs and physical cores.psutil docsGlossary counts them; recorded as n_threads
Split mode-sm-smThe llama.cpp option that sets the split mode: how a model is distributed across several GPUs. The layer mode assigns whole layers to each GPU, row splits the weight tensors of each layer across the GPUs, and none uses a single GPU.llama-bench READMEGlossarylayer
Flash attention-fa1 (on)
Logical batch size-b-bThe llama.cpp option that sets the logical batch size: the maximum number of prompt tokens passed to the model in one call. Each logical batch is split into physical batches of the size set by -ub, so a larger logical batch can process long prompts in fewer calls, at the cost of more memory.llama-bench READMEGlossary2,048
Physical batch size-ub-ubThe llama.cpp option that sets the physical batch size: the number of tokens computed at once within a logical batch (-b). It trades memory use against the speed of prompt processing.llama-bench READMEGlossary512
Repetitions-rNot set, so llama-bench's default applies; production output has five repetitions per line
Output-ojsonl (JSON LinesJSON LinesA text format that stores one JSON value, usually an object, per line, so a file can be written and read one record at a time. It suits logs and streamed results, and line-based tools such as grep can process it. Files in this format usually end in .jsonl.JSON LinesGlossary), printed to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary as is

GPU Layers​

With the CUDA build, this benchmark sets how many model layers to offload to the GPUs (-ngl-nglThe llama.cpp option that sets how many layers of a model are offloaded to the GPUs. The remaining layers run on the CPU, so a model that does not fit in GPU memory can still use the GPU in part, and a value higher than the model's number of layers, such as 999, offloads all of them.llama-bench READMEGlossary) before it measures each model. It tries 999, 40, 24, and 12, in this order. For each value, it runs llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary with one repetition, one thread, and a fixed timeout of 120 seconds, and uses the first value whose run succeeds. If none succeeds, or with the CPU build, it uses 0. The value is recorded as n_gpu_layers.

Timeouts and Early Stops​

Each measurement's timeout in seconds depends on the model size, the token count, and the expected speed for that task and token count:

timeout = round((model_GiB / 0.25 + 1 + 5 × tokens / expected_speed) × scale)

The model_GiB variable is the model file's size in GiB. The formula allows 4 seconds per GiB to load the model, 1 second of overhead, and five repetitions at the expected speed. The scale variable is the value of --benchmark-timeout-scale (default: 1). The expected speeds rise with the token count:

TokensPrompt processing (tokens/s)Text generation (tokens/s)
1621
128105
5122525
1,0245050
4,096250250
16,3841,000Not run

For example, text generation of 16 tokens with a model of 4 GiB gets 4 / 0.25 + 1 + 5 × 16 / 1 = 97 seconds, and at 512 tokens 119 seconds. A server that generates 15 tokens/s needs ~171 seconds for the five repetitions at 512 tokens, so its measurement times out and the larger token counts of that task are skipped. The load allowance adds some slack: a server slightly below the expected speed can still finish in time.

After a successful measurement of 1,024 or more tokens that is not the last of its task, this benchmark also estimates whether the next token count can meet its expected speed. If the measurement took longer than the following limit, the remaining token counts of that task are skipped:

limit = timeout × expected_speed / next_expected_speed × 1.1

The timeout and expected_speed variables are those of the measurement that just ran, and next_expected_speed is the expected speed of the next token count. For prompt processing at 1,024 tokens, that limit is 22% of the timeout (50 / 250 × 1.1).

When a measurement times out or fails, the following happens:

  • On the first, 16-token measurement of either task, this benchmark stops and skips all larger models.
  • On a later token count, the remaining token counts of that task are skipped for that model.

To allow slower servers more time, raise --benchmark-timeout-scale. See Missing Results for the other reasons a run can stop early and how Spare Cores Navigator shows them.