# Glossary

> The terms used in the docs of the nvbandwidth image.

Acronyms and terms used on this image's pages. See the [site-wide glossary](https://docs.sparecores.com/glossary.md) for all terms.

## C {#letter-c}

### CE {#ce}

**Copy Engine**: A DMA (Direct Memory Access) engine of an NVIDIA GPU that copies data between GPU memory, host memory, and the memory of other GPUs without using the GPU's compute cores. A GPU has one or more copy engines, which CUDA uses for memory copies, so data transfers can run at the same time as compute kernels. See [CUDA programming guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#asynchronous-concurrent-execution).

### CPU {#cpu}

**Central Processing Unit**: The general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads. See [Wikipedia](https://en.wikipedia.org/wiki/Central_processing_unit).

### CPU affinity {#cpu-affinity}

The set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread binding or CPU pinning. See [Linux man page](https://man7.org/linux/man-pages/man2/sched_setaffinity.2.html).

### CUDA {#cuda}

**Compute Unified Device Architecture**: The parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN. See [NVIDIA docs](https://docs.nvidia.com/cuda/).

### CUDA Toolkit {#cuda-toolkit}

NVIDIA's development kit for building CUDA programs, with the nvcc compiler, headers, and libraries such as the CUDA runtime and cuBLAS. A program is built against one toolkit version and runs on any host whose NVIDIA driver supports that CUDA version, so the toolkit used for the build and the CUDA version a host reports can differ. See [NVIDIA docs](https://docs.nvidia.com/cuda/).

## G {#letter-g}

### GPU {#gpu}

**Graphics Processing Unit**: A processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory. See [Wikipedia](https://en.wikipedia.org/wiki/Graphics_processing_unit).

## H {#letter-h}

### harness {#harness}

The code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures. See [Wikipedia](https://en.wikipedia.org/wiki/Test_harness).

## J {#letter-j}

### JSON {#json}

**JavaScript Object Notation**: A text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output. See [JSON](https://www.json.org/).

## L {#letter-l}

### LLM inference speed {#llm-inference-speed}

How fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generation, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.

## N {#letter-n}

### `nvbandwidth` {#nvbandwidth}

An open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers. See [GitHub](https://github.com/NVIDIA/nvbandwidth).

### nvbandwidth test case {#nvbandwidth-nvbandwidth-test-case}

One named measurement of the `nvbandwidth` tool, such as `host_to_device_memcpy_ce`, defined by its copy direction, the GPUs taking part, and its copy method, given by the `_ce` or `_sm` suffix. A test case between GPUs is waived rather than run when no two GPUs of the system have P2P (Peer-to-Peer) access to each other, as on a system with one GPU. The `nvbandwidth -l` command lists the test cases of a build. See [Official documentation](https://github.com/NVIDIA/nvbandwidth/blob/v0.8/README.md).

### NVIDIA Container Toolkit {#nvidia-container-toolkit}

A set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container. See [NVIDIA docs](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html).

### NVIDIA driver {#nvidia-driver}

The kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images. See [NVIDIA docs](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html).

### NVLink {#nvlink}

A high-bandwidth, point-to-point interconnect by NVIDIA that links GPUs to each other, and in some systems to the CPU, directly rather than through PCIe. Between GPUs in the same server, it gives several times the bandwidth of a PCIe link, and NVSwitch chips connect many GPUs over NVLink at once. See [NVIDIA](https://www.nvidia.com/en-us/data-center/nvlink/).

## P {#letter-p}

### P2P {#p2p}

**Peer-to-Peer**: A model in which devices or computers exchange data directly with each other instead of through a central server or host. In GPU computing, it means copies and memory access between two GPUs over NVLink or PCIe, without staging the data in host memory; whether two GPUs support it depends on the hardware and the driver. See [CUDA programming guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#peer-to-peer-memory-access).

### PCIe {#pcie}

**Peripheral Component Interconnect Express**: The standard high-speed serial bus that connects expansion devices, such as GPUs, NVMe SSDs, and network cards, to a computer's CPU. A link has 1 to 16 lanes, and each generation roughly doubles the bandwidth per lane. A PCIe 4.0 link with 16 lanes transfers about 32 GB/s in each direction. See [Wikipedia](https://en.wikipedia.org/wiki/PCI_Express).

## R {#letter-r}

### `resource-tracker` {#resource-tracker}

An open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload. See [resource-tracker on GitHub](https://github.com/SpareCores/resource-tracker-rs).

## S {#letter-s}

### SM {#sm}

**Streaming Multiprocessor**: The basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number. See [CUDA programming guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#hardware-implementation).

### stdout {#stdout}

The standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as `> results.jsonl`, while logs on stderr still reach the terminal. See [Wikipedia](https://en.wikipedia.org/wiki/Standard_streams).
