# Glossary

> The terms used in the docs of the vLLM manual.

Acronyms and terms used on this manual's pages. See the [site-wide glossary](https://docs.sparecores.com/glossary.md) for all terms.

## A {#letter-a}

### access token {#access-token}

A secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted. See [Hugging Face Hub docs](https://huggingface.co/docs/hub/security-tokens).

### AMD64 {#amd64}

The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it `amd64` or `x86_64`. See [Wikipedia](https://en.wikipedia.org/wiki/X86-64).

### Arm64 {#arm64}

The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it `arm64` or `aarch64`. See [Wikipedia](https://en.wikipedia.org/wiki/AArch64).

### attention head {#attention-head}

One of several parallel attention computations in a layer of a transformer model. Each head projects the tokens into its own smaller query, key, and value vectors and learns which earlier tokens to focus on, and the layer combines the outputs of all heads. The number of heads per layer is fixed by the model's architecture, and methods that split a layer across devices, such as tensor parallelism, usually assign whole heads to each device. See [Attention Is All You Need](https://arxiv.org/abs/1706.03762).

### AVX-512 {#avx-512}

**Advanced Vector Extensions 512**: A set of x86 SIMD instructions that operate on 512-bit registers, so that one instruction processes, for example, 16 single-precision floating-point numbers at once. Intel introduced it in its server CPUs with the Skylake generation, and AMD supports it from the Zen 4 generation. It is made of several subsets, such as the foundation, `avx512f`, and `avx512_bf16` for bfloat16 arithmetic, which Linux lists as flags in `/proc/cpuinfo`. See [Wikipedia](https://en.wikipedia.org/wiki/AVX-512).

### AVX2 {#avx2}

**Advanced Vector Extensions 2**: A set of x86 SIMD instructions that operate on 256-bit registers, introduced by Intel with the Haswell generation in 2013 and supported by AMD since 2015. It extends most integer vector instructions of the earlier AVX set to 256 bits. Nearly all current x86 CPUs support it, and AVX-512 is its wider successor. See [Wikipedia](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#Advanced_Vector_Extensions_2).

### AWS {#aws}

**Amazon Web Services**: The cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning. See [AWS](https://aws.amazon.com/).

## B {#letter-b}

### bfloat16 {#bfloat16}

A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively. See [Wikipedia](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format).

### `bitsandbytes` {#vllm-common-bitsandbytes}

An open-source Python library that quantizes the weights of PyTorch models to 8 or 4 bits, so that large models need less GPU memory. Hugging Face Transformers and vLLM can load models quantized with it, and its 4-bit format is also used to fine-tune large models on a single GPU. See [bitsandbytes on GitHub](https://github.com/bitsandbytes-foundation/bitsandbytes).

## C {#letter-c}

### CI {#ci}

**Continuous Integration**: The practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged. See [Wikipedia](https://en.wikipedia.org/wiki/Continuous_integration).

### compute capability {#compute-capability}

A version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later. See [NVIDIA](https://developer.nvidia.com/cuda-gpus).

### container entrypoint {#container-entrypoint}

The executable that a container image runs when a container starts, set by the `ENTRYPOINT` instruction of its Dockerfile. The image's default command (`CMD`), or the arguments given after the image name in `docker run`, are passed to it as arguments, so an entrypoint can wrap any command, for example to set up the environment or record resource usage first. The `docker run --entrypoint` option replaces it. See [Dockerfile reference](https://docs.docker.com/reference/dockerfile/#entrypoint).

### CPU {#cpu}

**Central Processing Unit**: The general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads. See [Wikipedia](https://en.wikipedia.org/wiki/Central_processing_unit).

### CUDA {#cuda}

**Compute Unified Device Architecture**: The parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN. See [NVIDIA docs](https://docs.nvidia.com/cuda/).

## D {#letter-d}

### data parallelism {#vllm-common-data-parallelism}

A way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices. See [Hugging Face docs](https://huggingface.co/docs/transformers/perf_train_gpu_many).

### Docker Hub {#docker-hub}

Docker's public registry of container images, where projects publish their images under a name and tag, such as `ubuntu:24.04`. Docker pulls images from it by default when a name gives no other registry. See [Docker docs](https://docs.docker.com/docker-hub/).

### `--dtype` {#vllm-common-dtype}

The vLLM option that sets the data type of the model weights and activations, such as `bfloat16` or `float32`. With its default, `auto`, vLLM derives the data type from the model's configuration. A smaller data type needs less memory and bandwidth per weight, but only runs fast on hardware that supports it. See [vLLM engine arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html).

## E {#letter-e}

### E2EL {#vllm-common-e2el}

**End-to-End Request Latency**: The time from sending a request to an LLM server until the last output token of its response arrives. It covers queuing, prompt processing, and the generation of every output token, so it is the time to first token plus the generation time of the further tokens, and it sets how long a full response takes. See [NVIDIA NIM docs](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html).

## F {#letter-f}

### float16 {#float16}

A 16-bit floating-point format, also called half precision, defined by the IEEE 754 standard. It has a 5-bit exponent and a 10-bit mantissa, so it takes half the memory of float32 but covers a much narrower range of values, up to 65,504. Machine learning frameworks use it to save memory and speed up computation. See [Wikipedia](https://en.wikipedia.org/wiki/Half-precision_floating-point_format).

### float32 {#float32}

A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory. See [Wikipedia](https://en.wikipedia.org/wiki/Single-precision_floating-point_format).

## G {#letter-g}

### gated model {#gated-model}

A model on the Hugging Face Hub whose files can be downloaded only after a logged-in user accepts the conditions set by its authors, such as a license. Programs that download it then need an access token of an account that has accepted them. See [Hugging Face docs](https://huggingface.co/docs/hub/models-gated).

### GPU {#gpu}

**Graphics Processing Unit**: A processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory. See [Wikipedia](https://en.wikipedia.org/wiki/Graphics_processing_unit).

### `--gpu-memory-utilization` {#vllm-common-gpu-memory-utilization}

The vLLM option that sets the fraction of device memory, from 0 to 1, that a vLLM worker may use for the model weights, intermediate results, and key-value cache. On GPUs, it applies to the memory of each GPU; on the CPU backend of vLLM, each worker applies it to the total memory of its NUMA node, despite the option's name. The memory left after the weights and intermediate results goes to the key-value cache, so a higher value lets the server hold more concurrent sequences. See [vLLM engine arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html).

### GuideLLM {#guidellm}

An open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports. See [GuideLLM on GitHub](https://github.com/vllm-project/guidellm).

### GuideLLM sweep {#vllm-common-guidellm-sweep}

A GuideLLM benchmark profile that measures a server across a range of loads. It first runs a synchronous stage, which sends one request at a time, and a throughput stage, which sends as many concurrent requests as allowed. It then runs the remaining stages, if any, at constant request rates between the rates of those two stages. The sweep size is the total number of stages. See [GuideLLM on GitHub](https://github.com/vllm-project/guidellm).

## H {#letter-h}

### harness {#harness}

The code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures. See [Wikipedia](https://en.wikipedia.org/wiki/Test_harness).

### HTTP {#http}

**Hypertext Transfer Protocol**: The request-response protocol of the web, in which a client asks a server for a resource by URL and gets back a status code, headers, and a body. HTTPS is HTTP over an encrypted connection. See [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP).

### Hugging Face {#hugging-face}

A company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models. See [Hugging Face Hub docs](https://huggingface.co/docs/hub).

### Hugging Face model ID {#vllm-common-hugging-face-model-id}

The name of a model repository on the Hugging Face Hub, in the form `<owner>/<model>`, such as `Qwen/Qwen2.5-0.5B-Instruct`. Programs and libraries use it to download the model's files, and it appears as the title of the model's page on the Hub. See [Hugging Face Hub docs](https://huggingface.co/docs/hub/models-the-hub).

## I {#letter-i}

### ITL {#vllm-common-itl}

**Inter-Token Latency**: The time between two consecutive output tokens of a response that an LLM server streams. It shows how smoothly text appears while it is generated, and is usually reported as percentiles or the mean over all gaps between tokens. See [NVIDIA NIM docs](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html).

## J {#letter-j}

### JSON {#json}

**JavaScript Object Notation**: A text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output. See [JSON](https://www.json.org/).

### JSON Lines {#json-lines}

A text format that stores one JSON value, usually an object, per line, so a file can be written and read one record at a time. It suits logs and streamed results, and line-based tools such as grep can process it. Files in this format usually end in `.jsonl`. See [JSON Lines](https://jsonlines.org/).

## K {#letter-k}

### key-value cache {#key-value-cache}

The memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks. See [Hugging Face docs](https://huggingface.co/docs/transformers/kv_cache).

### `--kv-cache-memory-bytes` {#vllm-common-kv-cache-memory-bytes}

The vLLM option that sets the size of the key-value cache of each worker in bytes. Without it, vLLM derives the cache size from `--gpu-memory-utilization`; this option gives finer control over how much memory the cache takes. See [vLLM engine arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html).

## L {#letter-l}

### LLM {#llm}

**Large Language Model**: A neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code. See [Wikipedia](https://en.wikipedia.org/wiki/Large_language_model).

### LLM inference speed {#llm-inference-speed}

How fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generation, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.

## M {#letter-m}

### `--max-model-len` {#vllm-common-max-model-len}

The vLLM option that sets the context length of a model, the maximum number of tokens, prompt and output together, in one sequence. The key-value cache must have room for at least one sequence of this length, so a shorter context length needs less memory. Without the option, vLLM uses the length from the model's configuration. See [vLLM engine arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html).

### `--max-num-batched-tokens` {#vllm-common-max-num-batched-tokens}

The vLLM option that sets the maximum number of tokens that a vLLM engine processes in one scheduling step, summed over all sequences in the batch. Prompt tokens and generated tokens both count toward it, so a lower value limits the work and memory of each step, and a higher value lets more prompt tokens be processed at once. See [vLLM engine arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html).

### `--max-num-seqs` {#vllm-common-max-num-seqs}

The vLLM option that sets the maximum number of sequences, or requests, that a vLLM engine processes together in one scheduling step. Further requests wait in a queue until a running sequence finishes. With data parallelism, each replica applies the limit on its own. See [vLLM engine arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html).

## N {#letter-n}

### NUMA {#numa}

**Non-Uniform Memory Access**: A memory design where each CPU socket or chiplet has its own local memory, which its cores reach faster than the memory of the others. Software that keeps its threads and their memory on the same NUMA node avoids the slower remote accesses. See [Linux kernel docs](https://docs.kernel.org/mm/numa.html).

### NVIDIA A10G {#nvidia-a10g}

A data center GPU of NVIDIA's Ampere generation with 24 GB of GDDR6 memory and compute capability 8.6, a variant of the NVIDIA A10 made for AWS. It powers the EC2 G5 instances, which are used for machine learning inference and graphics workloads. See [AWS](https://aws.amazon.com/ec2/instance-types/g5/).

### NVIDIA Container Toolkit {#nvidia-container-toolkit}

A set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container. See [NVIDIA docs](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html).

### NVIDIA driver {#nvidia-driver}

The kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images. See [NVIDIA docs](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html).

### NVIDIA L40S {#nvidia-l40s}

A data center GPU of NVIDIA's Ada Lovelace generation with 48 GB of GDDR6 memory and compute capability 8.9. It is built for AI inference and training as well as graphics and video workloads. See [NVIDIA](https://www.nvidia.com/en-us/data-center/l40s/).

## O {#letter-o}

### OpenMP {#openmp}

An API for shared-memory parallel programming in C, C++, and Fortran, made of compiler directives, library routines, and environment variables. A program marks loops or regions to run in parallel, and the OpenMP runtime splits them across a team of threads, which can be bound to specific CPU cores. See [OpenMP](https://www.openmp.org/).

## P {#letter-p}

### pipeline parallelism {#vllm-common-pipeline-parallelism}

A way to split a model across several devices by assigning consecutive groups of its layers to each, so that each device holds only part of the weights. The output of one stage passes to the next, and the stages work on different requests or micro-batches at the same time to keep all devices busy. It needs less communication between devices than tensor parallelism. See [Hugging Face docs](https://huggingface.co/docs/transformers/perf_train_gpu_many).

### `psutil` {#psutil}

A cross-platform Python library for retrieving information on running processes and system utilization, such as CPU, memory, disks, and network. It can count both logical CPUs and physical cores. See [psutil docs](https://psutil.readthedocs.io/).

## Q {#letter-q}

### quantization {#quantization}

Storing the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy. See [Hugging Face docs](https://huggingface.co/docs/optimum/concept_guides/quantization).

## R {#letter-r}

### RAM {#ram}

**Random-Access Memory**: The main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off. See [Wikipedia](https://en.wikipedia.org/wiki/Random-access_memory).

### `resource-tracker` {#resource-tracker}

An open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload. See [resource-tracker on GitHub](https://github.com/SpareCores/resource-tracker-rs).

### RPS {#rps}

**Requests per Second**: The number of requests a server completes per second over a measurement window. It measures a server's capacity in whole requests, independent of their size, so it is compared only between runs with the same request mix. See [NVIDIA NIM docs](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html).

## S {#letter-s}

### `SIGTERM` {#sigterm}

The termination signal of POSIX systems, which asks a process to exit cleanly. It is the default signal of the `kill` command, and `docker stop` sends it to a container's main process first, then SIGKILL, which cannot be caught, if the process has not exited after a grace period. By convention, a process that exits because of it returns 143 (128 plus 15). See [signal(7) man page](https://man7.org/linux/man-pages/man7/signal.7.html).

### SM {#sm}

**Streaming Multiprocessor**: The basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number. See [CUDA programming guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#hardware-implementation).

### SMT {#smt}

**Simultaneous Multithreading**: A CPU technique that runs two (or more) hardware threads on one physical core, sharing its execution units, e.g. Intel Hyper-Threading. It raises throughput when a thread stalls, but the threads compete for the core. Cloud vendors usually sell each thread as a vCPU. See [Wikipedia](https://en.wikipedia.org/wiki/Simultaneous_multithreading).

### stderr {#stderr}

The standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program. See [Wikipedia](https://en.wikipedia.org/wiki/Standard_streams).

### stdout {#stdout}

The standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as `> results.jsonl`, while logs on stderr still reach the terminal. See [Wikipedia](https://en.wikipedia.org/wiki/Standard_streams).

### swappiness {#swappiness}

A Linux kernel setting, `vm.swappiness`, that sets how readily the kernel swaps out process memory compared with dropping cached file data when it needs free memory. Higher values swap more eagerly; 0 makes the kernel avoid swapping as long as it can. Container runtimes such as Docker can also set it per container. See [Linux kernel docs](https://docs.kernel.org/admin-guide/sysctl/vm.html#swappiness).

## T {#letter-t}

### tensor parallelism {#tensor-parallelism}

A way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count. See [Hugging Face docs](https://huggingface.co/docs/transformers/perf_train_gpu_many).

### thread binding {#thread-binding}

Restricting each thread of a program to chosen CPU cores by setting its CPU affinity, also called CPU pinning. It keeps a thread's data in the caches and local memory of its cores, and stops the operating system from moving it between cores during a run, which makes performance faster and more repeatable, especially on hosts with several NUMA nodes. See [Linux man page](https://man7.org/linux/man-pages/man2/sched_setaffinity.2.html).

### token {#token}

The unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.

### TPOT {#vllm-common-tpot}

**Time per Output Token**: The average time an LLM server takes to generate each output token of a response after the first one, that is, the generation time after the first token divided by the number of further tokens. Together with the time to first token, it sets how long a full response takes. See [NVIDIA NIM docs](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html).

### TPS {#vllm-common-tps}

**Tokens per Second**: The number of tokens an LLM server processes or generates per second. Benchmarks count it in different ways, over output tokens only or over prompt and output tokens, per request or summed over all concurrent requests, so a TPS figure is compared only with one counted the same way. See [NVIDIA NIM docs](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html).

### TTFT {#ttft}

**Time to First Token**: The time from sending a request to an LLM server until the first output token arrives. It includes queuing, prompt processing, and the generation of the first token, so it grows with the prompt length and the server's load, and it sets how long a user waits before a response starts. See [NVIDIA NIM docs](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html).

## V {#letter-v}

### vCPU {#vcpu}

**virtual CPU**: A CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core. See [Wikipedia](https://en.wikipedia.org/wiki/Virtual_CPU).

### vLLM {#vllm}

**virtual Large Language Model**: An open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API. See [vLLM docs](https://docs.vllm.ai/), [vLLM on GitHub](https://github.com/vllm-project/vllm), [Origin of the name](https://github.com/vllm-project/vllm/issues/835).

### `VLLM_CPU_OMP_THREADS_BIND` {#vllm-common-vllm-cpu-omp-threads-bind}

The environment variable of vLLM's CPU backend that sets which CPU cores the OpenMP threads of each worker run on. It takes CPU lists such as `0-31`, one per worker, separated by vertical bars, so `0-31|32-63` binds two workers to 32 cores each. The value `auto` binds each worker to the cores of one NUMA node, and `nobind` turns the binding off. See [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu.html).
