Glossary
Acronyms and other terms used in these docs, with links to learn more. Hover a term where the docs spell it out, e.g. "CRFCRF: Constant Rate FactorThe quality-based rate control of the x264 and x265 encoders, which varies the bitrate to keep the perceived quality constant across scenes. Lower values mean higher quality and larger files; x264's default is 23 on a scale of 0 to 51.FFmpeg wikiGlossary (Constant Rate Factor)", or a license such as MPL 2.0MPL 2.0: Mozilla Public License 2.0The open-source license of the Navigator's software components. Anyone may use, modify and distribute the code, also commercially and combined with closed-source code, but changes to the licensed files must be published under the same license.Spare Cores licenseMozilla FAQGlossary, to see its entry here.
General
The terms used across the docs, with a tooltip on every page.
A
AAC
Advanced Audio Coding: A lossy audio codec standardized in MPEG-2 and MPEG-4 as the successor of MP3. It sounds better than MP3 at the same bitrate, and it is the usual audio track of MP4 files and streaming video. See FFmpeg AAC encoder.
access token
A secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted. See Hugging Face Hub docs.
AMD64
The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64. See Wikipedia.
API
Application Programming Interface: A defined set of requests and responses through which one program uses another, such as a web service's HTTP endpoints or a library's functions. See Wikipedia.
Arm64
The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64. See Wikipedia.
ASIC
Application-Specific Integrated Circuit: An integrated circuit designed for one specific task instead of general-purpose computing. It runs that task faster and with less energy than a CPU or GPU, but cannot be reprogrammed for other work. Examples are Bitcoin mining chips, network switch chips, and AI accelerators such as TPUs. See Wikipedia.
attention head
One of several parallel attention computations in a layer of a transformer model. Each head projects the tokens into its own smaller query, key, and value vectors and learns which earlier tokens to focus on, and the layer combines the outputs of all heads. The number of heads per layer is fixed by the model's architecture, and methods that split a layer across devices, such as tensor parallelism, usually assign whole heads to each device. See Attention Is All You Need.
AVX-512
Advanced Vector Extensions 512: A set of x86 SIMD instructions that operate on 512-bit registers, so that one instruction processes, for example, 16 single-precision floating-point numbers at once. Intel introduced it in its server CPUs with the Skylake generation, and AMD supports it from the Zen 4 generation. It is made of several subsets, such as the foundation, avx512f, and avx512_bf16 for bfloat16 arithmetic, which Linux lists as flags in /proc/cpuinfo. See Wikipedia.
AVX2
Advanced Vector Extensions 2: A set of x86 SIMD instructions that operate on 256-bit registers, introduced by Intel with the Haswell generation in 2013 and supported by AMD since 2015. It extends most integer vector instructions of the earlier AVX set to 256 bits. Nearly all current x86 CPUs support it, and AVX-512 is its wider successor. See Wikipedia.
AWS
Amazon Web Services: The cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning. See AWS.
B
bfloat16
A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively. See Wikipedia.
binserve
An open-source static web server written in Rust, distributed as a single binary and configured with one JSON file. It maps URL routes to files and directories, can keep files cached in memory, and can also serve encrypted connections and list directory contents. See binserve on GitHub.
blkdiscard
A Linux command from util-linux that discards (trims) a range of blocks, by default all blocks of a device, and so deletes their data. An SSD then treats those blocks as unwritten and can erase them in the background, which restores the write speed of a fresh drive. See man page.
brotli
A lossless compression format by Google, standardized in RFC 7932. It combines LZ77-style matching, Huffman coding, and context modeling with a built-in dictionary of common web text, and is widely used to compress HTTP responses. Its quality levels range from 0 (fastest) to 11 (smallest output). See RFC 7932.
BSL 1.1
Business Source License 1.1: The source-available license of the Navigator's Database snapshots. Anyone may copy, modify and redistribute them, and use them outside production. Production use is free for academic research and for organizations with less than USD 1 million in annual revenue that don't build a competing cloud data, benchmarking or price-comparison service; other production use needs a commercial license. Three years after its publication, each snapshot is also available under CC BY-SA 4.0CC BY-SA 4.0: Creative Commons Attribution-ShareAlike 4.0The open license of the data the Navigator collects, such as the Crawler Data and the Inspector Data. Anyone may copy, share and adapt the data, also commercially, if they credit Spare Cores, indicate their changes, and share their adaptations under the same license.Spare Cores licenseCreative Commons summaryGlossary. See Spare Cores license, MariaDB FAQ.
bw_mem
The memory bandwidth tool of LMbench. It moves data through a buffer of a given size in one of several modes, such as rd (read and sum an array), wr (write a constant to each element), rdwr (read, then write the same location), and the copy modes cp, fcp, and bcopy. It reports the buffer size in MB and the bandwidth in MB/s; with -P, it runs several processes, each with its own buffer, and reports their total bandwidth. See bw_mem manual.
bzip2
A lossless file compressor by Julian Seward, first released in 1996. It sorts each block of data with the Burrows-Wheeler transform, then encodes it with move-to-front and Huffman coding. It usually compresses better than gzip but runs slower; its levels 1 to 9 set the block size from 100 to 900 kB. See bzip2.
bzip3
A lossless compressor by Kamila Szewczyk that, like bzip2, is built on the Burrows-Wheeler transform, combined with context modeling. It typically compresses text better than bzip2, and its configurable block size trades memory use for compression ratio. See bzip3 on GitHub.
C
CC BY 3.0
Creative Commons Attribution 3.0: A Creative Commons license that allows anyone to copy, adapt and use a work, also commercially, as long as they credit its author. See License.
CC BY-SA 4.0
Creative Commons Attribution-ShareAlike 4.0: The open license of the data the Navigator collects, such as the Crawler Data and the Inspector Data. Anyone may copy, share and adapt the data, also commercially, if they credit Spare Cores, indicate their changes, and share their adaptations under the same license. See Spare Cores license, Creative Commons summary.
CC0 1.0
Creative Commons Zero 1.0: A Creative Commons dedication with which authors waive all their copyright in a work, placing it as close to the public domain as the law allows. Anyone may use the work for any purpose, without attribution. See License.
CDN
Content Delivery Network: A network of distributed servers that cache content, such as web pages, images or video, and serve it from locations close to its users. It lowers latency for the users and offloads traffic from the origin server. See Wikipedia.
CE
Copy Engine: A DMA (Direct Memory Access) engine of an NVIDIA GPU that copies data between GPU memory, host memory, and the memory of other GPUs without using the GPU's compute cores. A GPU has one or more copy engines, which CUDA uses for memory copies, so data transfers can run at the same time as compute kernels. See CUDA programming guide.
cgroup v2
control groups v2: The Linux kernel feature that organizes processes into a hierarchy of groups, and limits and accounts for the CPU, memory and I/O of each group. Container runtimes such as Docker use it to enforce the resource limits of a container. It replaced the original cgroup v1 with a single, unified hierarchy. See Linux kernel docs, Docker resource constraints.
CI
Continuous Integration: The practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged. See Wikipedia.
compression ratio
A measure of how much a compressor shrinks data, from the sizes of the uncompressed and the compressed data. It is most often given as uncompressed size divided by compressed size, where higher is better, but some tools report compressed size divided by uncompressed size as a percentage, where lower is better. See Wikipedia.
compute capability
A version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later. See NVIDIA.
container entrypoint
The executable that a container image runs when a container starts, set by the ENTRYPOINT instruction of its Dockerfile. The image's default command (CMD), or the arguments given after the image name in docker run, are passed to it as arguments, so an entrypoint can wrap any command, for example to set up the environment or record resource usage first. The docker run --entrypoint option replaces it. See Dockerfile reference.
CPU
Central Processing Unit: The general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads. See Wikipedia.
CPU affinity
The set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread binding or CPU pinning. See Linux man page.
CPU Mark
The overall CPU score of PassMark PerformanceTest. It combines the results of the CPU tests, such as integer and floating-point math, prime numbers, sorting, encryption, compression, physics, and a single-threaded test, into one unitless number, where higher is better. PassMark Software publishes CPU Mark results submitted by users for thousands of processors. See PassMark CPU Benchmarks.
CPU-credit throttling
A limit on burstable cloud server types, which earn CPU credits while they run below a baseline CPU usage and spend them to run above it. When the credits run out, the provider throttles the server to its baseline, so sustained load runs slower than a short burst. See AWS burstable instances.
CTE
Common Table Expression: A named temporary result set, defined with WITH at the start of a SQL statement, that the rest of the statement can query like a table. CTEs make complex queries easier to read, and recursive CTEs can walk hierarchies such as trees. See PostgreSQL docs.
CUDA
Compute Unified Device Architecture: The parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN. See NVIDIA docs.
CUDA Toolkit
NVIDIA's development kit for building CUDA programs, with the nvcc compiler, headers, and libraries such as the CUDA runtime and cuBLAS. A program is built against one toolkit version and runs on any host whose NVIDIA driver supports that CUDA version, so the toolkit used for the build and the CUDA version a host reports can differ. See NVIDIA docs.
CV
Coefficient of Variation: The standard deviation of a set of measurements divided by their mean, often given as a percentage. As it is scale-free, it compares the variability of measurements of different magnitudes, such as how much repeated runs of different benchmarks vary. See Wikipedia.
D
DBaaS
Database as a Service: A cloud service model in which the vendor runs a database for its users, handling provisioning, backups, patching and replication. Examples are Amazon RDS and Google Cloud SQL. See Wikipedia.
DMI
Desktop Management Interface: A standard from the 1990s for describing a computer's hardware to management software. Its data model lives on in SMBIOS, which is why the firmware tables that describe a server's hardware are often called DMI tables, and Linux exposes some of their values under /sys/class/dmi/id/. See Wikipedia.
Docker Hub
Docker's public registry of container images, where projects publish their images under a name and tag, such as ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry. See Docker docs.
DRAM
Dynamic Random-Access Memory: The most common type of volatile computer memory, which stores each bit as a charge in a capacitor that has to be refreshed periodically. Servers use it as main memory, and many storage devices as a cache. See Wikipedia.
F
flash attention
An exact attention algorithm for transformer models, published as FlashAttention in 2022, that computes attention in tiles held in fast on-chip GPU memory instead of writing the full attention matrix to the slower main GPU memory. It cuts memory traffic and memory use, which speeds up inference and training, especially with long inputs. LLM inference engines such as llama.cpp and vLLM offer it as an option. See FlashAttention paper.
float16
A 16-bit floating-point format, also called half precision, defined by the IEEE 754 standard. It has a 5-bit exponent and a 10-bit mantissa, so it takes half the memory of float32 but covers a much narrower range of values, up to 65,504. Machine learning frameworks use it to save memory and speed up computation. See Wikipedia.
float32
A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory. See Wikipedia.
fps
Frames Per Second: The number of frames, the individual images of an animation or simulation, that a program produces each second. A higher rate means smoother motion or faster processing. See Wikipedia.
G
gated model
A model on the Hugging Face Hub whose files can be downloaded only after a logged-in user accepts the conditions set by its authors, such as a license. Programs that download it then need an access token of an account that has accepted them. See Hugging Face docs.
GCP
Google Cloud Platform: The cloud computing platform of Google. It offers virtual machines (Compute Engine), object storage, managed databases, Kubernetes and machine learning services, among others. See Google Cloud.
Geekbench
A cross-platform CPU and GPU compute benchmark by Primate Labs. Its CPU benchmark runs workloads modeled on real applications, such as file compression, image processing, and web browsing, first on one core and then on all cores, and reports a Single-Core and a Multi-Core score. Geekbench 6 scores are calibrated so that a baseline system scores 2500. See Geekbench.
GGML
Georgi Gerganov Machine Learning: An open-source tensor library for machine learning, written in C and C++ without external dependencies. It runs on CPUs of many architectures, with SIMD-optimized code, and on GPUs and other accelerators through backends, and it supports quantized weights of 2 to 8 bits. llama.cpp and the GGUF file format are built on it. See ggml on GitHub.
GGUF
GGML Universal File: A binary file format for LLM weights and metadata, introduced in 2023 by the ggml project behind llama.cpp. One file holds the model's tensors, often quantized, together with its tokenizer and settings, so a model can be loaded quickly with memory mapping and without other files. Hugging Face hosts many models in this format. See GGUF specification, Hugging Face docs.
GHCR
GitHub Container Registry: GitHub's registry of container images, part of GitHub Packages, where images are published as ghcr.io/<owner>/<image>:<tag>. Each image gets a package page on GitHub that shows its description, a docker pull command, and the README of the repository it is linked to. See GitHub docs.
Google Brain
A deep learning research team at Google, founded in 2011. It developed the TensorFlow framework and the bfloat16 number format, and its researchers co-authored the paper that introduced the Transformer architecture, the basis of today's LLMs. In 2023, Google merged it with DeepMind into Google DeepMind. See Wikipedia.
GPU
Graphics Processing Unit: A processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory. See Wikipedia.
GuideLLM
An open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports. See GuideLLM on GitHub.
gzip
A lossless file compressor of the GNU Project, first released in 1992, and its file format, specified in RFC 1952. It uses the DEFLATE algorithm, which combines LZ77 matching and Huffman coding, and is the most widely supported compression format on Unix-like systems and the web. Its levels range from 1 (fastest) to 9 (smallest output). See GNU gzip.
H
H.264
A lossy video codec standardized in 2003, also called AVC. It is the most widely supported video format, used in Blu-ray, broadcast, streaming, and video calls. Common encoders are x264 and hardware encoders such as NVENC. See Specification.
H.265
Another name for HEVC, from the number of its standard. See Specification.
harness
The code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures. See Wikipedia.
HDD
Hard Disk Drive: A storage device that reads and writes data with magnetic heads on spinning platters. It offers cheap capacity, but each random access waits for a head to move and a platter to turn, so it completes far fewer random operations per second than an SSD. See Wikipedia.
HEVC
High Efficiency Video Coding: A lossy video codec standardized in 2013 as the successor of H.264, also called H.265. It compresses about twice as well as H.264 at the same quality, at the cost of slower encoding, and is used for 4K video and streaming. Common encoders are x265 and NVENC. See Specification.
HTTP
Hypertext Transfer Protocol: The request-response protocol of the web, in which a client asks a server for a resource by URL and gets back a status code, headers, and a body. HTTPS is HTTP over an encrypted connection. See MDN.
HTTPS
Hypertext Transfer Protocol Secure: HTTP, the protocol of the web, encrypted with TLS. It protects requests and responses from eavesdropping and tampering, and its certificates let clients verify the identity of the server. See Wikipedia.
Hugging Face
A company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models. See Hugging Face Hub docs.
I
I/O
Input/Output: The transfer of data between a computer and the outside world, such as reading and writing disks or sending and receiving network packets. A workload limited by its speed rather than by the CPU is called I/O-bound. See Wikipedia.
IaaS
Infrastructure as a Service: A cloud service model that rents out compute, storage and network resources, such as virtual machines and disks, on demand. The vendor runs the hardware, and the user manages the operating system and everything above it. See Wikipedia.
ID
identifier: A name or number that refers to exactly one item, such as a server type, a benchmark run, or a database row.
IETF
Internet Engineering Task Force: The open standards organization that develops the protocols of the internet, such as TCP/IP, HTTP, and TLS. It publishes its standards as numbered RFC (Request for Comments) documents. See IETF.
IOPS
Input/Output Operations Per Second: The number of read and write operations a storage device completes per second. It is the main measure of random-access performance, and depends on the block size, the queue depth and the mix of reads and writes. See Wikipedia.
J
JDBC
Java Database Connectivity: The standard Java API for connecting to relational databases and running SQL statements against them. Each database vendor supplies a JDBC driver, so one program can work with many databases through the same interface. See Java tutorial.
JIT
Just-in-Time Compilation: The compilation of a program's code to machine code while the program runs, instead of before. The compiler can use what is known only at run time, such as the actual data, but the compile time adds to the run time, so it pays off mostly on long-running code. PostgreSQL, for example, compiles parts of long, CPU-bound analytical queries, such as their expressions, with LLVM. See Wikipedia, PostgreSQL docs.
JSON
JavaScript Object Notation: A text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output. See JSON.
JSON Lines
A text format that stores one JSON value, usually an object, per line, so a file can be written and read one record at a time. It suits logs and streamed results, and line-based tools such as grep can process it. Files in this format usually end in .jsonl. See JSON Lines.
K
key-value cache
The memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks. See Hugging Face docs.
KVM
Kernel-based Virtual Machine: The virtualization module of the Linux kernel, which lets Linux run VMs with the hardware virtualization extensions of Intel and AMD CPUs. A server can run VMs with it only if its CPU exposes these extensions to the operating system, which many cloud VMs do not. See Wikipedia.
L
llama-bench
The benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines. See llama-bench README.
llama.cpp
An open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench. See llama.cpp on GitHub.
LLM
Large Language Model: A neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code. See Wikipedia.
LLM inference speed
How fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generation, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.
LMbench
A suite of portable micro-benchmarks for Unix-like systems, written by Larry McVoy and Carl Staelin. Its tools each measure one basic operation, such as memory bandwidth and latency, context switches, or system call overhead. See LMbench.
LVM
Logical Volume Manager: The Linux tool that pools disks or partitions into volume groups and divides these into logical volumes, which can be resized while in use. It also supports snapshots, and striping or mirroring across disks. See man page.
lz4
A lossless compression algorithm and format by Yann Collet from the LZ77 family, built for speed rather than compression ratio. It compresses and especially decompresses much faster than gzip, at a lower ratio; its higher levels use a slower high-compression mode that keeps decompression fast. See LZ4.
lzma
Lempel-Ziv-Markov chain Algorithm: A lossless compression algorithm by Igor Pavlov, used in the 7z and xz formats. It combines LZ77 matching over a large dictionary with range coding, reaching high compression ratios with slow compression and fast decompression. Its presets range from 0 to 9. See 7-Zip LZMA SDK.
M
Memory Mark
The overall memory score of PassMark PerformanceTest. It combines the results of the memory tests, such as database operations, cached and uncached reads, writes, and latency, into one unitless number, where higher is better. See PassMark Memory Benchmarks.
memtier_benchmark
An open-source load generator for Redis and Memcached, developed by Redis Ltd. It runs many concurrent connections over one or more threads, sends a configurable mix of SET and GET requests, and reports operations per second, latency averages, and latency percentiles per request type. See memtier_benchmark on GitHub.
MobileNet v1 SSD
A neural network for object detection in images. It combines MobileNet v1, a small convolutional network designed for mobile devices, with a Single Shot Detector, which finds and classifies all objects in one pass of the network, so it is light enough to run on phones and is often used to measure the speed of machine-learning inference. See MobileNets paper, SSD paper.
MP3
MPEG-1 Audio Layer III: A lossy audio codec and file format from the 1990s, still the most widely supported format for compressed music. See Wikipedia.
MP4
MPEG-4 Part 14: A container file format that holds video, audio, and subtitle tracks, most often H.264 or HEVC video with AAC audio. It is the common format for video on the web and on phones. See Wikipedia.
MPL 2.0
Mozilla Public License 2.0: The open-source license of the Navigator's software components. Anyone may use, modify and distribute the code, also commercially and combined with closed-source code, but changes to the licensed files must be published under the same license. See Spare Cores license, Mozilla FAQ.
N
nested paging
Hardware support for memory virtualization, in which the CPU translates a guest's virtual addresses through the guest's page tables and then through a second set of page tables kept by the hypervisor. It removes the need for the hypervisor to maintain shadow page tables, but a TLB miss then needs more memory lookups. Intel calls it Extended Page Tables and AMD Rapid Virtualization Indexing. See Wikipedia.
netem
Network Emulator: A Linux traffic control queueing discipline that adds delay, jitter, packet loss, duplication or reordering to the packets leaving a network interface. It is used to test how software behaves on slow or unreliable networks. See SRT cookbook.
nginx
An open-source web server, also used as a reverse proxy, load balancer, and HTTP cache. It handles many connections in a few worker processes, each running an event loop, and has many settings for workers, buffers, caching, and timeouts that are usually tuned per deployment. See nginx docs.
nice
A Unix command that runs a program with a changed scheduling priority, from -20 (highest) to 19 (lowest). Raising a process's priority above the default needs the CAP_SYS_NICE capability, which root has. See man page.
NTP
Network Time Protocol: A protocol that keeps computer clocks in sync with reference time servers over a network. A client adjusts its system clock in small steps or, when it is far off, in one jump, so wall-clock time on a server can move backward or skip ahead. See NTP.
NUMA
Non-Uniform Memory Access: A memory design where each CPU socket or chiplet has its own local memory, which its cores reach faster than the memory of the others. Software that keeps its threads and their memory on the same NUMA node avoids the slower remote accesses. See Linux kernel docs.
numactl
A Linux command that runs a program with a NUMA policy, such as binding it to the CPUs and memory of given NUMA nodes, and shows the NUMA layout of the host. See man page.
nvbandwidth
An open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers. See GitHub.
NVIDIA A10G
A data center GPU of NVIDIA's Ampere generation with 24 GB of GDDR6 memory and compute capability 8.6, a variant of the NVIDIA A10 made for AWS. It powers the EC2 G5 instances, which are used for machine learning inference and graphics workloads. See AWS.
NVIDIA Container Toolkit
A set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container. See NVIDIA docs.
NVIDIA driver
The kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images. See NVIDIA docs.
NVIDIA L40S
A data center GPU of NVIDIA's Ada Lovelace generation with 48 GB of GDDR6 memory and compute capability 8.9. It is built for AI inference and training as well as graphics and video workloads. See NVIDIA.
NVIDIA T4G
A data center GPU of NVIDIA's Turing generation with 16 GB of GDDR6 memory and compute capability 7.5, a variant of the NVIDIA T4 made for AWS. It powers the EC2 G5g instances, which pair it with Arm-based AWS Graviton2 CPUs. See AWS.
NVLink
A high-bandwidth, point-to-point interconnect by NVIDIA that links GPUs to each other, and in some systems to the CPU, directly rather than through PCIe. Between GPUs in the same server, it gives several times the bandwidth of a PCIe link, and NVSwitch chips connect many GPUs over NVLink at once. See NVIDIA.
NVMe
Non-Volatile Memory Express: The interface specification for accessing non-volatile storage, most often SSDs, over PCI Express. It is designed for the parallelism of flash memory, with many deep command queues, and NVMe over Fabrics extends it to network transports. See NVM Express.
O
OLTP
Online Transaction Processing: A class of workloads made of many short, concurrent transactions that each read or update a few rows, such as orders, payments or bookings. OLTP databases are optimized for the latency and throughput of such transactions, unlike analytical databases built for large scans. See Wikipedia.
OpenMP
An API for shared-memory parallel programming in C, C++, and Fortran, made of compiler directives, library routines, and environment variables. A program marks loops or regions to run in parallel, and the OpenMP runtime splits them across a team of threads, which can be bound to specific CPU cores. See OpenMP.
OS
operating system: The software that manages a computer's hardware and runs programs on it, such as Linux or Windows. A cloud server runs the OS of the image it was started from.
out-of-memory killer
The Linux kernel mechanism that frees memory when the system, or a cgroup, runs out of it and cannot reclaim enough. It picks a process by a badness score, mostly based on its memory use, and kills it. Container runtimes rely on it to enforce memory limits. See Linux kernel docs.
OVHcloud
A French cloud provider, founded in 1999, with data centers in Europe, North America and Asia-Pacific. It offers dedicated (bare-metal) servers, public cloud instances based on OpenStack, and private cloud. See OVHcloud.
P
P2P
Peer-to-Peer: A model in which devices or computers exchange data directly with each other instead of through a central server or host. In GPU computing, it means copies and memory access between two GPUs over NVLink or PCIe, without staging the data in host memory; whether two GPUs support it depends on the hardware and the driver. See CUDA programming guide.
PassMark
The PerformanceTest benchmark suite of PassMark Software, for Windows, Linux, macOS, and mobile devices. It runs fixed CPU, memory, disk, and graphics tests, such as math workloads involving compression, encryption, and physics simulations, and combines the results into an overall PassMark Rating. See PerformanceTest.
PCIe
Peripheral Component Interconnect Express: The standard high-speed serial bus that connects expansion devices, such as GPUs, NVMe SSDs, and network cards, to a computer's CPU. A link has 1 to 16 lanes, and each generation roughly doubles the bandwidth per lane. A PCIe 4.0 link with 16 lanes transfers about 32 GB/s in each direction. See Wikipedia.
pgbench
The benchmark tool shipped with PostgreSQL. It runs a SQL script repeatedly over many concurrent client sessions, and reports the throughput in transactions per second and the latency. Its built-in script is modeled on TPC-B, and custom scripts can define any other workload. See PostgreSQL docs.
PID
Process Identifier: The number by which the operating system identifies a running process. On Linux, a PID is assigned when a process starts and can be reused after it exits, and commands such as kill use it to target a process. See Wikipedia.
PostgreSQL
An open-source object-relational database system, developed by a global community and descended from the POSTGRES project at UC Berkeley in 1986. It uses and extends SQL, with transactions, many index types, extensions, and JSON data. See PostgreSQL docs.
prompt processing
The first phase of LLM inference, also called prefill, in which the model reads all tokens of the input prompt and stores their intermediate results in its key-value cache. The prompt's tokens are processed together in batches, so this phase is usually limited by compute and runs much faster per token than text generation. Its speed decides how long a long input takes before the first output token appears.
psutil
A cross-platform Python library for retrieving information on running processes and system utilization, such as CPU, memory, disks, and network. It can count both logical CPUs and physical cores. See psutil docs.
PyPI
Python Package Index: The official public repository of third-party Python packages, run by the Python Software Foundation. Tools such as pip download and install packages from it by default. See PyPI.
Q
quantization
Storing the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy. See Hugging Face docs.
R
RAID
Redundant Array of Independent Disks: A way to combine several disks into one logical device, for redundancy, speed or both. Common levels are RAID 0 (striping), RAID 1 (mirroring) and RAID 5 or 6 (striping with parity); on Linux, the md driver implements RAID in software. See Wikipedia.
RAM
Random-Access Memory: The main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off. See Wikipedia.
RAM user
A user identity in Alibaba Cloud's Resource Access Management service, with its own credentials and permissions. Programs and people use RAM users to call the cloud's APIs instead of the main account, so each gets only the permissions it needs. See Alibaba Cloud docs.
RDBMS
Relational Database Management System: A database system that stores data in tables of rows and columns, related through keys and queried with SQL. It keeps data consistent with constraints and transactions; PostgreSQL, MySQL and Oracle Database are examples. See Wikipedia.
Redis
An open-source, in-memory key-value data store, used as a database, cache, and message broker. It keeps its data in RAM, supports data structures such as strings, hashes, lists, and sets, and can persist data to disk with periodic snapshots or an append-only log. See Redis docs.
resource-tracker
An open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload. See resource-tracker on GitHub.
RPS
Requests per Second: The number of requests a server completes per second over a measurement window. It measures a server's capacity in whole requests, independent of their size, so it is compared only between runs with the same request mix. See NVIDIA NIM docs.
RTT
Round-Trip Time: The time a network packet takes to reach its destination plus the time its response takes to get back. It sets a lower bound on the latency of every request-response exchange, and grows with the distance between the endpoints. See Wikipedia.
S
S3
Simple Storage Service: The object storage service of AWS, which stores files as objects in buckets and serves them over HTTP. Its API has become a de facto standard that many other storage services and tools support. See Amazon S3.
SCore
Spare Cores score: A CPU speed score of a server type in Spare Cores Navigator, used to compare the speed of server types. It is measured with the cpu stressor of stress-ng at increasing worker counts. The single-core SCore is the result with one worker; the multi-core SCore, the raw multi-core CPU speed that SCore means on its own, is the highest result.
SHA-256
Secure Hash Algorithm 256: A cryptographic hash function of the SHA-2 family that maps any data to a 256-bit digest. Even a one-bit change in the data gives a completely different digest, so it is used to verify that a downloaded file is intact, and in digital signatures and certificates. See NIST FIPS 180-4.
shared-core
Describes a cloud server type whose vCPUs share physical cores with other servers, often with a guaranteed baseline of only a fraction of a core. Such server types are cheaper, but their CPU performance can vary with the load of neighboring servers or be limited by CPU credits. See Google Cloud shared-core machine types.
SIGINT
The interrupt signal of POSIX systems, which a terminal sends to the running process when the user presses Ctrl+C. A process can catch it to clean up before it exits; by convention, a process that exits because of a signal returns 128 plus the signal number, 130 for SIGINT. See signal(7) man page.
SIGTERM
The termination signal of POSIX systems, which asks a process to exit cleanly. It is the default signal of the kill command, and docker stop sends it to a container's main process first, then SIGKILL, which cannot be caught, if the process has not exited after a grace period. By convention, a process that exits because of it returns 143 (128 plus 15). See signal(7) man page.
Silesia corpus
A set of 12 files, ~211 MB in total, compiled by Sebastian Deorowicz at the Silesian University of Technology to test lossless compression on a mix of data types: text, executables, databases, images, and XML. Its dickens file holds the collected works of Charles Dickens, ~10 MB of English text. See Silesia corpus.
SIMD
Single Instruction, Multiple Data: A form of parallel processing in which one CPU instruction applies the same operation to several data elements at once. CPUs implement it with vector instruction set extensions, which x86 and Arm CPUs both provide. See Wikipedia.
SM
Streaming Multiprocessor: The basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number. See CUDA programming guide.
SMART
Self-Monitoring, Analysis, and Reporting Technology: A monitoring system built into HDDs and SSDs that records health and error counters, such as reallocated sectors, power-on hours or wear level. Tools such as smartctl read these counters to spot failing disks early. See Wikipedia.
smartctl
The command-line tool of the smartmontools package that reads SMART data, self-test logs and device information from ATA, SCSI and NVMe disks, and can start self-tests. See smartmontools.
SMBIOS
System Management BIOS: A standard for the tables in which a computer's firmware describes its hardware to the operating system, such as the system's manufacturer and model, its CPUs and memory modules, and the firmware version. It grew out of DMI, and on Linux, tools such as dmidecode read these tables. In a VM, the virtualization platform sets the values. See DMTF SMBIOS standard.
SMT
Simultaneous Multithreading: A CPU technique that runs two (or more) hardware threads on one physical core, sharing its execution units, e.g. Intel Hyper-Threading. It raises throughput when a thread stalls, but the threads compete for the core. Cloud vendors usually sell each thread as a vCPU. See Wikipedia.
SQL
Structured Query Language: The standard language to define, query and modify data in relational databases. Its declarative statements, such as SELECT, INSERT and UPDATE, describe what data to return or change, and the database plans how to do it. See Wikipedia.
SSD
Solid-State Drive: A storage device that keeps data in flash memory instead of on spinning disks, so it reads and writes much faster than an HDD, especially at random locations. A controller with its own firmware maps logical blocks to flash pages, and many SSDs write incoming data to a fast cache first, then move it to slower flash. Cloud servers use SSDs as local disks and behind network block storage. See Wikipedia.
SSL
Secure Sockets Layer: A protocol that encrypts network connections and verifies the identity of the server, and optionally of the client, with certificates. TLS has replaced it, but SSL is still the common name for both, including in PostgreSQL's settings. See PostgreSQL docs.
stderr
The standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program. See Wikipedia.
stdout
The standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal. See Wikipedia.
stress-ng
An open-source tool that loads a computer in selectable ways, with hundreds of stress tests (stressors) that exercise the CPU, memory, file systems, and kernel interfaces. With --metrics, it reports each stressor's throughput in bogo ops/s. See stress-ng on GitHub.
swappiness
A Linux kernel setting, vm.swappiness, that sets how readily the kernel swaps out process memory compared with dropping cached file data when it needs free memory. Higher values swap more eagerly; 0 makes the kernel avoid swapping as long as it can. Container runtimes such as Docker can also set it per container. See Linux kernel docs.
T
tensor parallelism
A way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count. See Hugging Face docs.
text generation
The second phase of LLM inference, also called decoding, in which the model produces output tokens one at a time, each depending on the tokens before it. Every step reads all model weights and the key-value cache, so this phase is usually limited by memory bandwidth rather than compute, and runs much slower per token than prompt processing.
thread binding
Restricting each thread of a program to chosen CPU cores by setting its CPU affinity, also called CPU pinning. It keeps a thread's data in the caches and local memory of its cores, and stops the operating system from moving it between cores during a run, which makes performance faster and more repeatable, especially on hosts with several NUMA nodes. See Linux man page.
TLB
Translation Lookaside Buffer: A CPU cache of recent translations from virtual to physical memory addresses. A TLB shootdown invalidates entries on other cores after a mapping changes, which needs an interrupt to each of them. See Wikipedia.
token
The unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.
TPC-B
Transaction Processing Performance Council Benchmark B: A database benchmark of short, update-heavy banking transactions, each updating an account, a teller and a branch balance and logging the change. The TPC retired it in 1995, but the default script of pgbench is still modeled on it. See TPC.
TPU
Tensor Processing Unit: An ASIC designed by Google to accelerate machine learning workloads, built around hardware for the matrix multiplications of neural networks. It was first deployed in Google's data centers in 2015, and is offered as a service on Google Cloud. See Google Cloud docs.
TTFT
Time to First Token: The time from sending a request to an LLM server until the first output token arrives. It includes queuing, prompt processing, and the generation of the first token, so it grows with the prompt length and the server's load, and it sets how long a user waits before a response starts. See NVIDIA NIM docs.
U
URL
Uniform Resource Locator: The address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.
UTC
Coordinated Universal Time: The primary time standard by which the world sets its clocks, the successor of Greenwich Mean Time. It has no time zone offset or daylight saving time, so UTC timestamps compare directly across regions. See Wikipedia.
V
vCPU
virtual CPU: A CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core. See Wikipedia.
vLLM
virtual Large Language Model: An open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API. See vLLM docs, vLLM on GitHub, Origin of the name.
vllm serve
The vLLM command that starts an HTTP server for a model, with an API compatible with OpenAI's, such as the /v1/completions and /v1/chat/completions endpoints. It loads the model on the available devices, batches concurrent requests, and answers on a /health endpoint once it is ready. See vLLM docs.
VM
virtual machine: A computer emulated in software on a physical host, with its own OS and a share of the host's CPUs, memory, and storage. Most cloud servers are VMs. See Wikipedia.
VM exit
A transition from a virtual machine's guest code to the hypervisor, triggered by an operation the hypervisor has to handle, such as some privileged instructions, I/O, or interrupts. Each exit and return costs CPU time, so workloads that cause many exits run slower in a VM.
VPC
Virtual Private Cloud: An isolated virtual network within a public cloud account, with its own IP address ranges, subnets, routing and firewall rules. Resources such as virtual machines and databases are placed in it to control how they reach each other and the internet. See Wikipedia.
W
WAL
Write-Ahead Logging: A technique in which a database writes each change to a durable log before it changes the data files. After a crash, replaying the log restores a consistent state, and shipping it to other servers drives replication and point-in-time recovery. PostgreSQL, for example, calls its log the WAL, and on write-heavy workloads the speed of WAL writes often limits throughput. See Wikipedia, PostgreSQL docs.
wrk
An open-source HTTP benchmarking tool written in C. It generates load from a few threads, each handling many open connections through an event loop, and reports latency and requests-per-second statistics. Lua scripts can customize the requests it sends and process the responses. See wrk on GitHub.
X
XML
Extensible Markup Language: A text format for structured documents and data, built from nested, named elements in angle brackets. See W3C.
Y
YAML
YAML Ain't Markup Language: A human-readable text format for structured data, built from indented key-value mappings and lists. It is a superset of JSON, and is widely used for configuration files and for exchanging data between programs. See YAML specification.
YCSB
Yahoo! Cloud Serving Benchmark: An open-source benchmark framework, released by Yahoo! in 2010, for key-value and NoSQL databases. It runs standard workloads that mix reads, updates, inserts and scans in different proportions, and is widely used to compare cloud data stores. See Wikipedia.
Z
Zipfian
A skewed probability distribution, named after Zipf's law, in which an item's frequency is inversely proportional to its rank, so a few items are accessed far more often than the rest. Benchmarks use it to model real-world hot spots, such as popular products or users. See Wikipedia.
zpaq
A lossless compressor and archive format by Matt Mahoney, based on context mixing, which predicts each bit from many models and combines their predictions. It reaches very high compression ratios, but compresses and decompresses slowly, at about the same speed in both directions. Its methods range from 1 (fastest) to 5 (smallest output). See ZPAQ.
zram
A Linux kernel module that creates compressed block devices in memory. It is most often used as swap, so swapped-out pages are compressed in RAM instead of written to disk, trading CPU time for memory. See Linux kernel docs.
zstd
Zstandard: A lossless compression algorithm and format by Yann Collet at Facebook, released in 2016 and specified in RFC 8878. It combines LZ77 matching with fast entropy coding, and its levels, from 1 to 22, cover a wide range of trade-offs, from faster than gzip to ratios close to lzma. Decompression stays fast at every level. See Zstandard on GitHub.
benchmark
The terms of the benchmark image's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
Geekbench Browser
The online database of Geekbench results run by Primate Labs. The free version of Geekbench uploads every result there, where it is public; the paid Pro version can also keep results offline. See Geekbench Browser.
openssl speed
The OpenSSL command that measures the throughput of cryptographic algorithms, such as hashes and ciphers. It runs each algorithm for a fixed time on each of several block sizes and reports the bytes processed per second. With -multi, it runs several processes in parallel and reports their total. See openssl speed manual.
benchmark-ffmpeg
The terms of the benchmark-ffmpeg image's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
AV1
AOMedia Video 1: A royalty-free video codec by the Alliance for Open Media, the successor of VP9. It compresses better than H.264H.264A lossy video codec standardized in 2003, also called AVC. It is the most widely supported video format, used in Blu-ray, broadcast, streaming, and video calls. Common encoders are x264 and hardware encoders such as NVENC.SpecificationGlossary and HEVC at the same quality, at the cost of slower encoding. Common encoders are libaom, SVT-AV1 and rav1e. See Specification.
CQ
Constant Quality: The quality-based rate control of NVIDIA's NVENC hardware encoder, which varies the bitrate to keep a target quality level. Lower values mean higher quality and larger files, as with CRF in software encoders. See NVENC rate control.
CRF
Constant Rate Factor: The quality-based rate control of the x264 and x265 encoders, which varies the bitrate to keep the perceived quality constant across scenes. Lower values mean higher quality and larger files; x264's default is 23 on a scale of 0 to 51. See FFmpeg wiki.
FFmpeg
A free, open-source collection of libraries and command-line tools to decode, encode, transcode, filter and stream audio and video. Its ffmpeg command converts between nearly all media formats, and its libraries, such as libavcodec, power many media players and services. See ffmpeg docs.
FLAC
Free Lossless Audio Codec: A lossless audio codec by the Xiph.Org Foundation. It compresses audio without discarding any information, so decoding restores the original samples bit for bit. See FLAC.
null muxer
An FFmpeg output format, selected with -f null, that discards everything it receives instead of writing a file. It is used to measure decoding, filtering or encoding speed, or to run analysis filters, without the cost of writing to disk. See FFmpeg formats.
NVDEC
NVIDIA Video Decoder: The hardware video decoder of NVIDIA GPUs, a dedicated unit next to the CUDA cores. It decodes codecs such as H.264, HEVC, VP9 and, on newer GPUs, AV1, leaving the CPU and the CUDA cores free. See NVIDIA GPUs in FFmpeg.
NVENC
NVIDIA Video Encoder: The hardware video encoder of NVIDIA GPUs, a dedicated unit next to the CUDA cores. It encodes H.264, HEVC and, on newer GPUs, AV1, much faster than software encoders, though usually at a somewhat lower quality for the same bitrate. FFmpeg exposes it as encoders such as h264_nvenc. See NVIDIA GPUs in FFmpeg.
Opus
audio codec: A royalty-free, lossy audio codec standardized by the IETF in 2012. It covers speech and music from low to high bitrates with low delay, which makes it common in voice and video calls and web streaming. See Opus.
PSNR
Peak Signal-to-Noise Ratio: A quality metric that compares a processed image or video to its source by the mean squared error of their pixels, in decibels. Higher values mean a closer match. It is simple and widely used, but follows perceived quality only loosely. See FFmpeg filter.
SSIM
Structural Similarity Index Measure: A quality metric that compares a processed image or video to its source by local patterns of luminance, contrast and structure. A score of 1 means identical, and the score follows perceived quality more closely than PSNR. See FFmpeg filter.
Tears of Steel
A 2012 science-fiction short film by the Blender Foundation, made with open-source tools and released under CC BY 3.0. Its high-resolution footage is freely available, which makes it common test material for video codecs. See Project page.
VMAF
Video Multimethod Assessment Fusion: A perceptual video quality metric by Netflix that combines several elementary metrics with a machine learning model trained on human ratings. It scores from 0 to 100, and is widely used to tune and compare video encoders. See GitHub.
Vorbis
A royalty-free, lossy audio codec by the Xiph.Org Foundation, usually stored in Ogg files. It was designed as a free alternative to MP3MP3: MPEG-1 Audio Layer IIIA lossy audio codec and file format from the 1990s, still the most widely supported format for compressed music.WikipediaGlossary and AAC, and is used in games and streaming; Opus has largely replaced it for new uses. See Vorbis.
benchmark-llm
The terms of the benchmark-llm image's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
-b
The llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary option that sets the logical batch size: the maximum number of prompt tokens passed to the model in one call. Each logical batch is split into physical batches of the size set by -ub, so a larger logical batch can process long prompts in fewer calls, at the cost of more memory. See llama-bench README.
-ngl
The llama.cpp option that sets how many layers of a model are offloaded to the GPUs. The remaining layers run on the CPU, so a model that does not fit in GPU memory can still use the GPU in part, and a value higher than the model's number of layers, such as 999, offloads all of them. See llama-bench README.
Q4_K_M
A llama.cpp quantizationquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary type of the k-quant family. It stores most weights with about 4 bits each, in blocks that share scale factors, and keeps some tensors at higher precision. The M (medium) variant is a common compromise between file size and output quality. See Hugging Face GGUF docs.
-sm
The llama.cpp option that sets the split mode: how a model is distributed across several GPUs. The layer mode assigns whole layers to each GPU, row splits the weight tensors of each layer across the GPUs, and none uses a single GPU. See llama-bench README.
-ub
The llama.cpp option that sets the physical batch size: the number of tokens computed at once within a logical batch (-b). It trades memory use against the speed of prompt processing. See llama-bench README.
benchmark-passmark
The terms of the benchmark-passmark image's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
Kops/s
Thousand Operations/s: The throughput unit of the PassMark database operations test: thousands of operations per second, where higher is better.
Kstrings/s
Thousand Strings/s: The throughput unit of the PassMark string sorting test: thousands of strings sorted per second, where higher is better.
Mmat/s
Million Matrices/s: The throughput unit of the PassMark extended instructions test: millions of matrices processed per second, where higher is better.
Mnums/s
Million Primes/s: The throughput unit of the PassMark prime numbers test: millions of primes per second, where higher is better.
Mops/s
Million Operations/s: The throughput unit of the PassMark integer math, floating-point math, and single-threaded tests: millions of operations per second, where higher is better.
pt_linux
PerformanceTest for Linux: The command-line edition of PassMark PerformanceTest for Linux on x86-64 and Arm64 processors. It runs the CPU and memory tests of PerformanceTest in a terminal interface, can run them unattended, and saves the results to a YAMLYAML: YAML Ain't Markup LanguageA human-readable text format for structured data, built from indented key-value mappings and lists. It is a superset of JSON, and is widely used for configuration files and for exchanging data between programs.YAML specificationGlossary file. See PerformanceTest Linux.
benchmark-pgbench-postgres
The terms of the benchmark-pgbench-postgres image's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
BenchBase
A framework that benchmarks SQL databases, the successor to OLTPBench. It runs workloads such as YCSB and a Wikipedia workload against any relational database with a JDBCJDBC: Java Database ConnectivityThe standard Java API for connecting to relational databases and running SQL statements against them. Each database vendor supplies a JDBC driver, so one program can work with many databases through the same interface.Java tutorialGlossary (Java Database Connectivity) driver, and collects per-transaction-type latency and throughput logs. See GitHub.
BRIN
Block Range Index: A PostgreSQL index type that stores only a summary, such as the minimum and maximum value, of each range of consecutive table blocks. It is tiny compared to a B-tree, and works well on large tables whose values follow their physical order, such as timestamps of appended rows. See PostgreSQL docs.
concurrency points
The client counts at which a database benchmark measures throughput and latency, each with that number of connections open for a fixed time. Measuring several concurrency points, from one client up to a count derived from the server's size, shows how throughput and latency change with load.
GIN
Generalized Inverted Index: A PostgreSQL index type that maps each element of a composite value, such as an array item, a key of a JSON document, or a word of a text, to the rows that contain it. It speeds up containment and full-text searches, at the cost of slower writes than a B-tree. See PostgreSQL docs.
GUC
Grand Unified Configuration: The configuration system of PostgreSQL, and by extension its individual settings, such as shared_buffers or work_mem. A setting can be set in postgresql.conf, on the command line, per database or role, or per session with SET. See PostgreSQL docs.
HammerDB
An open-source database benchmark engine. It runs standardized workloads, such as TPROC-C, with the same test definitions on SQL Server, Db2, Oracle, PostgreSQL, MariaDB, and MySQL, so that results are comparable across database systems. Runs can be scripted and repeated automatically. See HammerDB.
Parallel Query
A PostgreSQL feature that splits one query across several worker processes, for example to scan, join or aggregate a large table on several CPU cores at once. The max_parallel_workers_per_gather setting limits how many workers one part of a query can use. See PostgreSQL docs.
pgtune
A web tool that suggests PostgreSQL settings, such as shared_buffers and work_mem, from the memory, CPU count, storage type and workload of a server. Its suggestions are a starting point for tuning rather than an optimum. See pgtune.
PL/pgSQL
PostgreSQL Procedural Language: The built-in procedural language of PostgreSQL, which adds variables, loops, conditionals and error handling to SQL. It is used to write functions, procedures and triggers that run inside the database server. See PostgreSQL docs.
sysbench
A scriptable, multi-threaded benchmark tool built on LuaJIT. Its tests cover CPU, memory, file I/O, threads, and mutexes, and its Lua scripts run transactional database workloads against MySQL and PostgreSQL. It reports rates and latency percentiles. See GitHub.
TOAST
The Oversized-Attribute Storage Technique: The mechanism with which PostgreSQL stores values too large for a regular 8 kB page, by compressing them, moving them out of line into a separate table, or both. It applies automatically to large values, such as long texts or JSON documents. See PostgreSQL docs.
TPM
Transactions Per Minute: A throughput measure of transaction processing systems: the number of transactions they complete per minute. TPC-C reports a variant of it, tpmC, which counts only new-order transactions, while tools such as pgbench report transactions per second (TPS), which is 1/60 of TPM. See Wikipedia.
TPROC-C
HammerDB's transaction processing workload: The OLTP workload of the HammerDB benchmark tool, derived from the TPC-C specification of an order-entry system with warehouses, orders and payments. It is not an official TPC-C implementation, so its results are not comparable with published TPC-C results. See HammerDB docs.
benchmark-redis
The terms of the benchmark-redis image's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
Redis pipelining
A technique in which a client sends several RedisRedisAn open-source, in-memory key-value data store, used as a database, cache, and message broker. It keeps its data in RAM, supports data structures such as strings, hashes, lists, and sets, and can persist data to disk with periodic snapshots or an append-only log.Redis docsGlossary commands without waiting for the reply to each one, then reads all replies together. It reduces the number of network round trips and system calls per command, which raises throughput, while each command waits longer for its reply. See Redis docs.
Redis SET command
The Redis command that stores a string value under a key, replacing any existing value. Options can set an expiry time, or store the value only if the key does or does not exist yet. See Redis docs.
benchmark-storage
The terms of the benchmark-storage image's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
fio
The Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency. See fio docs.
FTL
Flash Translation Layer: The firmware layer of an SSDSSD: Solid-State DriveA storage device that keeps data in flash memory instead of on spinning disks, so it reads and writes much faster than an HDD, especially at random locations. A controller with its own firmware maps logical blocks to flash pages, and many SSDs write incoming data to a fast cache first, then move it to slower flash. Cloud servers use SSDs as local disks and behind network block storage.WikipediaGlossary controller that maps the logical blocks the host sees to physical flash pages. Because flash has to be erased in large blocks before it is rewritten, the FTL writes new data to free pages, and runs garbage collection and wear leveling in the background. Wikipedia.
garbage collection
The background work in which an SSD frees flash for new writes. Flash can only be erased in large blocks, so the drive copies the still-valid pages out of a block, then erases the whole block. It runs when spare area runs low, and it slows the writes that arrive meanwhile, which is why an SSD's sustained write speed is lower than its burst speed. Programming languages use the same term for reclaiming unused memory. See Wikipedia.
SLC
Single-Level Cell: Flash memory that stores one bit per cell, which makes it faster and more durable than flash storing more bits per cell, such as TLC or QLC. Many SSDs run part of their flash in SLC mode as a write cache, and write more slowly once a long write fills it. See Wikipedia.
nvbandwidth
The terms of the nvbandwidth image's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
nvbandwidth test case
One named measurement of the nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary tool, such as host_to_device_memcpy_ce, defined by its copy direction, the GPUs taking part, and its copy method, given by the _ce or _sm suffix. A test case between GPUs is waived rather than run when no two GPUs of the system have P2PP2P: Peer-to-PeerA model in which devices or computers exchange data directly with each other instead of through a central server or host. In GPU computing, it means copies and memory access between two GPUs over NVLink or PCIe, without staging the data in host memory; whether two GPUs support it depends on the hardware and the driver.CUDA programming guideGlossary (Peer-to-Peer) access to each other, as on a system with one GPU. The nvbandwidth -l command lists the test cases of a build. See Official documentation.
stress-ng
The terms of the stress-ng image's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
stress-ng bogo ops/s
bogus operations per second: The throughput metric of stress-ng: how many times a stressor completes its own unit of work per second. Each stressor defines that unit differently, so the values compare only within one stressor and one stress-ng version. See stress-ng man page.
stress-ng div16 method
A method of the stress-ng cpu stressor, selected with --cpu-method div16, in which each operation runs 50,000 16-bit unsigned integer divisions. Integer division takes many CPU cycles and depends little on memory, so it loads the core's arithmetic units directly. See stress-ng man page.
stress-ng metrc line
A line in which stress-ng --metrics prints a stressor's results when the run ends, tagged metrc: the bogo ops count, the real, user, and system time, and the bogo ops/s, among other columns. See stress-ng man page.
stress-ng stressor
One stress test of stress-ng, such as cpu, memcpy, or fork, selected with a command-line option of the same name and run by one or more worker processes. See stress-ng man page.
vllm-common
The terms of the vllm-common manual's references.md, with a tooltip on its pages and on the pages of the images that link to that file. Its own glossary lists all acronyms and terms its pages use.
bitsandbytes
An open-source Python library that quantizes the weights of PyTorch models to 8 or 4 bits, so that large models need less GPU memory. Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary Transformers and vLLM can load models quantizedquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary with it, and its 4-bit format is also used to fine-tune large models on a single GPU. See bitsandbytes on GitHub.
data parallelism
A way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices. See Hugging Face docs.
--dtype
The vLLM option that sets the data type of the model weights and activations, such as bfloat16 or float32. With its default, auto, vLLM derives the data type from the model's configuration. A smaller data type needs less memory and bandwidth per weight, but only runs fast on hardware that supports it. See vLLM engine arguments.
E2EL
End-to-End Request Latency: The time from sending a request to an LLM server until the last output token of its response arrives. It covers queuing, prompt processing, and the generation of every output token, so it is the time to first token plus the generation time of the further tokens, and it sets how long a full response takes. See NVIDIA NIM docs.
--gpu-memory-utilization
The vLLM option that sets the fraction of device memory, from 0 to 1, that a vLLM worker may use for the model weights, intermediate results, and key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary. On GPUs, it applies to the memory of each GPU; on the CPU backend of vLLM, each worker applies it to the total memory of its NUMA node, despite the option's name. The memory left after the weights and intermediate results goes to the key-value cache, so a higher value lets the server hold more concurrent sequences. See vLLM engine arguments.
GuideLLM sweep
A GuideLLM benchmark profile that measures a server across a range of loads. It first runs a synchronous stage, which sends one request at a time, and a throughput stage, which sends as many concurrent requests as allowed. It then runs the remaining stages, if any, at constant request rates between the rates of those two stages. The sweep size is the total number of stages. See GuideLLM on GitHub.
Hugging Face model ID
The name of a model repository on the Hugging Face Hub, in the form <owner>/<model>, such as Qwen/Qwen2.5-0.5B-Instruct. Programs and libraries use it to download the model's files, and it appears as the title of the model's page on the Hub. See Hugging Face Hub docs.
ITL
Inter-Token Latency: The time between two consecutive output tokens of a response that an LLM server streams. It shows how smoothly text appears while it is generated, and is usually reported as percentiles or the mean over all gaps between tokens. See NVIDIA NIM docs.
--kv-cache-memory-bytes
The vLLM option that sets the size of the key-value cache of each worker in bytes. Without it, vLLM derives the cache size from --gpu-memory-utilization; this option gives finer control over how much memory the cache takes. See vLLM engine arguments.
--max-model-len
The vLLM option that sets the context length of a model, the maximum number of tokens, prompt and output together, in one sequence. The key-value cache must have room for at least one sequence of this length, so a shorter context length needs less memory. Without the option, vLLM uses the length from the model's configuration. See vLLM engine arguments.
--max-num-batched-tokens
The vLLM option that sets the maximum number of tokens that a vLLM engine processes in one scheduling step, summed over all sequences in the batch. Prompt tokens and generated tokens both count toward it, so a lower value limits the work and memory of each step, and a higher value lets more prompt tokens be processed at once. See vLLM engine arguments.
--max-num-seqs
The vLLM option that sets the maximum number of sequences, or requests, that a vLLM engine processes together in one scheduling step. Further requests wait in a queue until a running sequence finishes. With data parallelism, each replica applies the limit on its own. See vLLM engine arguments.
pipeline parallelism
A way to split a model across several devices by assigning consecutive groups of its layers to each, so that each device holds only part of the weights. The output of one stage passes to the next, and the stages work on different requests or micro-batches at the same time to keep all devices busy. It needs less communication between devices than tensor parallelismtensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary. See Hugging Face docs.
TPOT
Time per Output Token: The average time an LLM server takes to generate each output token of a response after the first one, that is, the generation time after the first token divided by the number of further tokens. Together with the time to first token, it sets how long a full response takes. See NVIDIA NIM docs.
TPS
Tokens per Second: The number of tokens an LLM server processes or generates per second. Benchmarks count it in different ways, over output tokens only or over prompt and output tokens, per request or summed over all concurrent requests, so a TPS figure is compared only with one counted the same way. See NVIDIA NIM docs.
VLLM_CPU_OMP_THREADS_BIND
The environment variable of vLLM's CPU backend that sets which CPU cores the OpenMPOpenMPAn API for shared-memory parallel programming in C, C++, and Fortran, made of compiler directives, library routines, and environment variables. A program marks loops or regions to run in parallel, and the OpenMP runtime splits them across a team of threads, which can be bound to specific CPU cores.OpenMPGlossary threads of each worker run on. It takes CPU lists such as 0-31, one per worker, separated by vertical bars, so 0-31|32-63 binds two workers to 32 cores each. The value auto binds each worker to the cores of one NUMA node, and nobind turns the binding off. See vLLM CPU installation.