Skip to main content

Glossary

Acronyms and terms used on this image's pages. See the site-wide glossary for all terms.

C​

CE​

Copy Engine: A DMA (Direct Memory Access) engine of an NVIDIA GPU that copies data between GPU memory, host memory, and the memory of other GPUs without using the GPU's compute cores. A GPU has one or more copy engines, which CUDA uses for memory copies, so data transfers can run at the same time as compute kernels. See CUDA programming guide.

CPU​

Central Processing Unit: The general-purpose processor of a computer, which runs the operating systemOS: operating systemThe software that manages a computer's hardware and runs programs on it, such as Linux or Windows. A cloud server runs the OS of the image it was started from.Glossary and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads. See Wikipedia.

CPU affinity​

The set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread bindingthread bindingRestricting each thread of a program to chosen CPU cores by setting its CPU affinity, also called CPU pinning. It keeps a thread's data in the caches and local memory of its cores, and stops the operating system from moving it between cores during a run, which makes performance faster and more repeatable, especially on hosts with several NUMA nodes.Linux man pageGlossary or CPU pinning. See Linux man page.

CUDA​

Compute Unified Device Architecture: The parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN. See NVIDIA docs.

CUDA Toolkit​

NVIDIA's development kit for building CUDA programs, with the nvcc compiler, headers, and libraries such as the CUDA runtime and cuBLAS. A program is built against one toolkit version and runs on any host whose NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary supports that CUDA version, so the toolkit used for the build and the CUDA version a host reports can differ. See NVIDIA docs.

G​

GPU​

Graphics Processing Unit: A processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory. See Wikipedia.

H​

harness​

The code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures. See Wikipedia.

J​

JSON​

JavaScript Object Notation: A text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for APIAPI: Application Programming InterfaceA defined set of requests and responses through which one program uses another, such as a web service's HTTP endpoints or a library's functions.WikipediaGlossary responses and benchmark output. See JSON.

L​

LLM inference speed​

How fast hardware runs an LLMLLM: Large Language ModelA neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code.WikipediaGlossary, usually measured in tokenstokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary per second. It is measured separately for prompt processingprompt processingThe first phase of LLM inference, also called prefill, in which the model reads all tokens of the input prompt and stores their intermediate results in its key-value cache. The prompt's tokens are processed together in batches, so this phase is usually limited by compute and runs much faster per token than text generation. Its speed decides how long a long input takes before the first output token appears.Glossary, in which the model reads the input prompt, and text generationtext generationThe second phase of LLM inference, also called decoding, in which the model produces output tokens one at a time, each depending on the tokens before it. Every step reads all model weights and the key-value cache, so this phase is usually limited by memory bandwidth rather than compute, and runs much slower per token than prompt processing.Glossary, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.

N​

nvbandwidth​

An open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers. See GitHub.

nvbandwidth test case​

One named measurement of the nvbandwidth tool, such as host_to_device_memcpy_ce, defined by its copy direction, the GPUs taking part, and its copy method, given by the _ce or _sm suffix. A test case between GPUs is waived rather than run when no two GPUs of the system have P2PP2P: Peer-to-PeerA model in which devices or computers exchange data directly with each other instead of through a central server or host. In GPU computing, it means copies and memory access between two GPUs over NVLink or PCIe, without staging the data in host memory; whether two GPUs support it depends on the hardware and the driver.CUDA programming guideGlossary (Peer-to-Peer) access to each other, as on a system with one GPU. The nvbandwidth -l command lists the test cases of a build. See Official documentation.

NVIDIA Container Toolkit​

A set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container. See NVIDIA docs.

NVIDIA driver​

The kernel module and libraries that let the operating systemOS: operating systemThe software that manages a computer's hardware and runs programs on it, such as Linux or Windows. A cloud server runs the OS of the image it was started from.Glossary and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images. See NVIDIA docs.

A high-bandwidth, point-to-point interconnect by NVIDIA that links GPUs to each other, and in some systems to the CPU, directly rather than through PCIe. Between GPUs in the same server, it gives several times the bandwidth of a PCIe link, and NVSwitch chips connect many GPUs over NVLink at once. See NVIDIA.

P​

P2P​

Peer-to-Peer: A model in which devices or computers exchange data directly with each other instead of through a central server or host. In GPU computing, it means copies and memory access between two GPUs over NVLinkNVLinkA high-bandwidth, point-to-point interconnect by NVIDIA that links GPUs to each other, and in some systems to the CPU, directly rather than through PCIe. Between GPUs in the same server, it gives several times the bandwidth of a PCIe link, and NVSwitch chips connect many GPUs over NVLink at once.NVIDIAGlossary or PCIe, without staging the data in host memory; whether two GPUs support it depends on the hardware and the driver. See CUDA programming guide.

PCIe​

Peripheral Component Interconnect Express: The standard high-speed serial bus that connects expansion devices, such as GPUs, NVMe SSDsSSD: Solid-State DriveA storage device that keeps data in flash memory instead of on spinning disks, so it reads and writes much faster than an HDD, especially at random locations. A controller with its own firmware maps logical blocks to flash pages, and many SSDs write incoming data to a fast cache first, then move it to slower flash. Cloud servers use SSDs as local disks and behind network block storage.WikipediaGlossary, and network cards, to a computer's CPU. A link has 1 to 16 lanes, and each generation roughly doubles the bandwidth per lane. A PCIe 4.0 link with 16 lanes transfers about 32 GB/s in each direction. See Wikipedia.

R​

resource-tracker​

An open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload. See resource-tracker on GitHub.

S​

SM​

Streaming Multiprocessor: The basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number. See CUDA programming guide.

stdout​

The standard output stream of a process, one of the three standard streams with stdin and stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal. See Wikipedia.