Skip to main content

Glossary

Acronyms and terms used on this manual's pages. See the site-wide glossary for all terms.

A​

access token​

A secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary read token, for example, lets a program download the gated modelsgated modelA model on the Hugging Face Hub whose files can be downloaded only after a logged-in user accepts the conditions set by its authors, such as a license. Programs that download it then need an access token of an account that has accepted them.Hugging Face docsGlossary whose licenses the account has accepted. See Hugging Face Hub docs.

AMD64​

The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64. See Wikipedia.

Arm64​

The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64. See Wikipedia.

attention head​

One of several parallel attention computations in a layer of a transformer model. Each head projects the tokens into its own smaller query, key, and value vectors and learns which earlier tokens to focus on, and the layer combines the outputs of all heads. The number of heads per layer is fixed by the model's architecture, and methods that split a layer across devices, such as tensor parallelismtensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary, usually assign whole heads to each device. See Attention Is All You Need.

AVX-512​

Advanced Vector Extensions 512: A set of x86 SIMD instructions that operate on 512-bit registers, so that one instruction processes, for example, 16 single-precision floating-point numbers at once. Intel introduced it in its server CPUs with the Skylake generation, and AMD supports it from the Zen 4 generation. It is made of several subsets, such as the foundation, avx512f, and avx512_bf16 for bfloat16bfloat16A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively.WikipediaGlossary arithmetic, which Linux lists as flags in /proc/cpuinfo. See Wikipedia.

AVX2​

Advanced Vector Extensions 2: A set of x86 SIMD instructions that operate on 256-bit registers, introduced by Intel with the Haswell generation in 2013 and supported by AMD since 2015. It extends most integer vector instructions of the earlier AVX set to 256 bits. Nearly all current x86 CPUs support it, and AVX-512 is its wider successor. See Wikipedia.

AWS​

Amazon Web Services: The cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning. See AWS.

B​

bfloat16​

A 16-bit floating-point format designed for machine learning by Google BrainGoogle BrainA deep learning research team at Google, founded in 2011. It developed the TensorFlow framework and the bfloat16 number format, and its researchers co-authored the paper that introduced the Transformer architecture, the basis of today's LLMs. In 2023, Google merged it with DeepMind into Google DeepMind.WikipediaGlossary, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32float32A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory.WikipediaGlossary but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively. See Wikipedia.

bitsandbytes​

An open-source Python library that quantizes the weights of PyTorch models to 8 or 4 bits, so that large models need less GPU memory. Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary Transformers and vLLM can load models quantizedquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary with it, and its 4-bit format is also used to fine-tune large models on a single GPU. See bitsandbytes on GitHub.

C​

CI​

Continuous Integration: The practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged. See Wikipedia.

compute capability​

A version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later. See NVIDIA.

container entrypoint​

The executable that a container image runs when a container starts, set by the ENTRYPOINT instruction of its Dockerfile. The image's default command (CMD), or the arguments given after the image name in docker run, are passed to it as arguments, so an entrypoint can wrap any command, for example to set up the environment or record resource usage first. The docker run --entrypoint option replaces it. See Dockerfile reference.

CPU​

Central Processing Unit: The general-purpose processor of a computer, which runs the operating systemOS: operating systemThe software that manages a computer's hardware and runs programs on it, such as Linux or Windows. A cloud server runs the OS of the image it was started from.Glossary and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads. See Wikipedia.

CUDA​

Compute Unified Device Architecture: The parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN. See NVIDIA docs.

D​

data parallelism​

A way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelismpipeline parallelismA way to split a model across several devices by assigning consecutive groups of its layers to each, so that each device holds only part of the weights. The output of one stage passes to the next, and the stages work on different requests or micro-batches at the same time to keep all devices busy. It needs less communication between devices than tensor parallelism.Hugging Face docsGlossary, which split one model across devices. See Hugging Face docs.

Docker Hub​

Docker's public registry of container images, where projects publish their images under a name and tag, such as ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry. See Docker docs.

--dtype​

The vLLM option that sets the data type of the model weights and activations, such as bfloat16 or float32. With its default, auto, vLLM derives the data type from the model's configuration. A smaller data type needs less memory and bandwidth per weight, but only runs fast on hardware that supports it. See vLLM engine arguments.

E​

E2EL​

End-to-End Request Latency: The time from sending a request to an LLM server until the last output token of its response arrives. It covers queuing, prompt processingprompt processingThe first phase of LLM inference, also called prefill, in which the model reads all tokens of the input prompt and stores their intermediate results in its key-value cache. The prompt's tokens are processed together in batches, so this phase is usually limited by compute and runs much faster per token than text generation. Its speed decides how long a long input takes before the first output token appears.Glossary, and the generation of every output token, so it is the time to first token plus the generation time of the further tokens, and it sets how long a full response takes. See NVIDIA NIM docs.

F​

float16​

A 16-bit floating-point format, also called half precision, defined by the IEEE 754 standard. It has a 5-bit exponent and a 10-bit mantissa, so it takes half the memory of float32 but covers a much narrower range of values, up to 65,504. Machine learning frameworks use it to save memory and speed up computation. See Wikipedia.

float32​

A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16bfloat16A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively.WikipediaGlossary trade some of its precision for less memory. See Wikipedia.

G​

gated model​

A model on the Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary Hub whose files can be downloaded only after a logged-in user accepts the conditions set by its authors, such as a license. Programs that download it then need an access tokenaccess tokenA secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted.Hugging Face Hub docsGlossary of an account that has accepted them. See Hugging Face docs.

GPU​

Graphics Processing Unit: A processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory. See Wikipedia.

--gpu-memory-utilization​

The vLLM option that sets the fraction of device memory, from 0 to 1, that a vLLM worker may use for the model weights, intermediate results, and key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary. On GPUs, it applies to the memory of each GPU; on the CPU backend of vLLM, each worker applies it to the total memory of its NUMA node, despite the option's name. The memory left after the weights and intermediate results goes to the key-value cache, so a higher value lets the server hold more concurrent sequences. See vLLM engine arguments.

GuideLLM​

An open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP APIAPI: Application Programming InterfaceA defined set of requests and responses through which one program uses another, such as a web service's HTTP endpoints or a library's functions.WikipediaGlossary, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports. See GuideLLM on GitHub.

GuideLLM sweep​

A GuideLLM benchmark profile that measures a server across a range of loads. It first runs a synchronous stage, which sends one request at a time, and a throughput stage, which sends as many concurrent requests as allowed. It then runs the remaining stages, if any, at constant request rates between the rates of those two stages. The sweep size is the total number of stages. See GuideLLM on GitHub.

H​

harness​

The code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures. See Wikipedia.

HTTP​

Hypertext Transfer Protocol: The request-response protocol of the web, in which a client asks a server for a resource by URLURL: Uniform Resource LocatorThe address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.Glossary and gets back a status code, headers, and a body. HTTPSHTTPS: Hypertext Transfer Protocol SecureHTTP, the protocol of the web, encrypted with TLS. It protects requests and responses from eavesdropping and tampering, and its certificates let clients verify the identity of the server.WikipediaGlossary is HTTP over an encrypted connection. See MDN.

Hugging Face​

A company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated modelsgated modelA model on the Hugging Face Hub whose files can be downloaded only after a logged-in user accepts the conditions set by its authors, such as a license. Programs that download it then need an access token of an account that has accepted them.Hugging Face docsGlossary need an access tokenaccess tokenA secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted.Hugging Face Hub docsGlossary. Its Python libraries, such as Transformers, load and run the models. See Hugging Face Hub docs.

Hugging Face model ID​

The name of a model repository on the Hugging Face Hub, in the form <owner>/<model>, such as Qwen/Qwen2.5-0.5B-Instruct. Programs and libraries use it to download the model's files, and it appears as the title of the model's page on the Hub. See Hugging Face Hub docs.

I​

ITL​

Inter-Token Latency: The time between two consecutive output tokens of a response that an LLM server streams. It shows how smoothly text appears while it is generated, and is usually reported as percentiles or the mean over all gaps between tokens. See NVIDIA NIM docs.

J​

JSON​

JavaScript Object Notation: A text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output. See JSON.

JSON Lines​

A text format that stores one JSON value, usually an object, per line, so a file can be written and read one record at a time. It suits logs and streamed results, and line-based tools such as grep can process it. Files in this format usually end in .jsonl. See JSON Lines.

K​

key-value cache​

The memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks. See Hugging Face docs.

--kv-cache-memory-bytes​

The vLLM option that sets the size of the key-value cache of each worker in bytes. Without it, vLLM derives the cache size from --gpu-memory-utilization--gpu-memory-utilizationThe vLLM option that sets the fraction of device memory, from 0 to 1, that a vLLM worker may use for the model weights, intermediate results, and key-value cache. On GPUs, it applies to the memory of each GPU; on the CPU backend of vLLM, each worker applies it to the total memory of its NUMA node, despite the option's name. The memory left after the weights and intermediate results goes to the key-value cache, so a higher value lets the server hold more concurrent sequences.vLLM engine argumentsGlossary; this option gives finer control over how much memory the cache takes. See vLLM engine arguments.

L​

LLM​

Large Language Model: A neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code. See Wikipedia.

LLM inference speed​

How fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generationtext generationThe second phase of LLM inference, also called decoding, in which the model produces output tokens one at a time, each depending on the tokens before it. Every step reads all model weights and the key-value cache, so this phase is usually limited by memory bandwidth rather than compute, and runs much slower per token than prompt processing.Glossary, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.

M​

--max-model-len​

The vLLM option that sets the context length of a model, the maximum number of tokens, prompt and output together, in one sequence. The key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary must have room for at least one sequence of this length, so a shorter context length needs less memory. Without the option, vLLM uses the length from the model's configuration. See vLLM engine arguments.

--max-num-batched-tokens​

The vLLM option that sets the maximum number of tokens that a vLLM engine processes in one scheduling step, summed over all sequences in the batch. Prompt tokens and generated tokens both count toward it, so a lower value limits the work and memory of each step, and a higher value lets more prompt tokens be processed at once. See vLLM engine arguments.

--max-num-seqs​

The vLLM option that sets the maximum number of sequences, or requests, that a vLLM engine processes together in one scheduling step. Further requests wait in a queue until a running sequence finishes. With data parallelismdata parallelismA way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices.Hugging Face docsGlossary, each replica applies the limit on its own. See vLLM engine arguments.

N​

NUMA​

Non-Uniform Memory Access: A memory design where each CPU socket or chiplet has its own local memory, which its cores reach faster than the memory of the others. Software that keeps its threads and their memory on the same NUMA node avoids the slower remote accesses. See Linux kernel docs.

NVIDIA A10G​

A data center GPU of NVIDIA's Ampere generation with 24 GB of GDDR6 memory and compute capabilitycompute capabilityA version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later.NVIDIAGlossary 8.6, a variant of the NVIDIA A10 made for AWS. It powers the EC2 G5 instances, which are used for machine learning inference and graphics workloads. See AWS.

NVIDIA Container Toolkit​

A set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container. See NVIDIA docs.

NVIDIA driver​

The kernel module and libraries that let the operating systemOS: operating systemThe software that manages a computer's hardware and runs programs on it, such as Linux or Windows. A cloud server runs the OS of the image it was started from.Glossary and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images. See NVIDIA docs.

NVIDIA L40S​

A data center GPU of NVIDIA's Ada Lovelace generation with 48 GB of GDDR6 memory and compute capability 8.9. It is built for AI inference and training as well as graphics and video workloads. See NVIDIA.

O​

OpenMP​

An API for shared-memory parallel programming in C, C++, and Fortran, made of compiler directives, library routines, and environment variables. A program marks loops or regions to run in parallel, and the OpenMP runtime splits them across a team of threads, which can be bound to specific CPU cores. See OpenMP.

P​

pipeline parallelism​

A way to split a model across several devices by assigning consecutive groups of its layers to each, so that each device holds only part of the weights. The output of one stage passes to the next, and the stages work on different requests or micro-batches at the same time to keep all devices busy. It needs less communication between devices than tensor parallelismtensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary. See Hugging Face docs.

psutil​

A cross-platform Python library for retrieving information on running processes and system utilization, such as CPU, memory, disks, and network. It can count both logical CPUs and physical cores. See psutil docs.

Q​

quantization​

Storing the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy. See Hugging Face docs.

R​

RAM​

Random-Access Memory: The main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off. See Wikipedia.

resource-tracker​

An open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload. See resource-tracker on GitHub.

RPS​

Requests per Second: The number of requests a server completes per second over a measurement window. It measures a server's capacity in whole requests, independent of their size, so it is compared only between runs with the same request mix. See NVIDIA NIM docs.

S​

SIGTERM​

The termination signal of POSIX systems, which asks a process to exit cleanly. It is the default signal of the kill command, and docker stop sends it to a container's main process first, then SIGKILL, which cannot be caught, if the process has not exited after a grace period. By convention, a process that exits because of it returns 143 (128 plus 15). See signal(7) man page.

SM​

Streaming Multiprocessor: The basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number. See CUDA programming guide.

SMT​

Simultaneous Multithreading: A CPU technique that runs two (or more) hardware threads on one physical core, sharing its execution units, e.g. Intel Hyper-Threading. It raises throughput when a thread stalls, but the threads compete for the core. Cloud vendors usually sell each thread as a vCPU. See Wikipedia.

stderr​

The standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program. See Wikipedia.

stdout​

The standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal. See Wikipedia.

swappiness​

A Linux kernel setting, vm.swappiness, that sets how readily the kernel swaps out process memory compared with dropping cached file data when it needs free memory. Higher values swap more eagerly; 0 makes the kernel avoid swapping as long as it can. Container runtimes such as Docker can also set it per container. See Linux kernel docs.

T​

tensor parallelism​

A way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLinkNVLinkA high-bandwidth, point-to-point interconnect by NVIDIA that links GPUs to each other, and in some systems to the CPU, directly rather than through PCIe. Between GPUs in the same server, it gives several times the bandwidth of a PCIe link, and NVSwitch chips connect many GPUs over NVLink at once.NVIDIAGlossary. The number of devices usually has to divide dimensions of the model, such as its attention headattention headOne of several parallel attention computations in a layer of a transformer model. Each head projects the tokens into its own smaller query, key, and value vectors and learns which earlier tokens to focus on, and the layer combines the outputs of all heads. The number of heads per layer is fixed by the model's architecture, and methods that split a layer across devices, such as tensor parallelism, usually assign whole heads to each device.Attention Is All You NeedGlossary count. See Hugging Face docs.

thread binding​

Restricting each thread of a program to chosen CPU cores by setting its CPU affinityCPU affinityThe set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread binding or CPU pinning.Linux man pageGlossary, also called CPU pinning. It keeps a thread's data in the caches and local memory of its cores, and stops the operating systemOS: operating systemThe software that manages a computer's hardware and runs programs on it, such as Linux or Windows. A cloud server runs the OS of the image it was started from.Glossary from moving it between cores during a run, which makes performance faster and more repeatable, especially on hosts with several NUMA nodes. See Linux man page.

token​

The unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.

TPOT​

Time per Output Token: The average time an LLM server takes to generate each output token of a response after the first one, that is, the generation time after the first token divided by the number of further tokens. Together with the time to first token, it sets how long a full response takes. See NVIDIA NIM docs.

TPS​

Tokens per Second: The number of tokens an LLM server processes or generates per second. Benchmarks count it in different ways, over output tokens only or over prompt and output tokens, per request or summed over all concurrent requests, so a TPS figure is compared only with one counted the same way. See NVIDIA NIM docs.

TTFT​

Time to First Token: The time from sending a request to an LLM server until the first output token arrives. It includes queuing, prompt processing, and the generation of the first token, so it grows with the prompt length and the server's load, and it sets how long a user waits before a response starts. See NVIDIA NIM docs.

V​

vCPU​

virtual CPU: A CPU of a virtual machineVM: virtual machineA computer emulated in software on a physical host, with its own OS and a share of the host's CPUs, memory, and storage. Most cloud servers are VMs.WikipediaGlossary, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core. See Wikipedia.

vLLM​

virtual Large Language Model: An open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary in fixed-size blocks, as an OSOS: operating systemThe software that manages a computer's hardware and runs programs on it, such as Linux or Windows. A cloud server runs the OS of the image it was started from.Glossary pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API. See vLLM docs, vLLM on GitHub, Origin of the name.

VLLM_CPU_OMP_THREADS_BIND​

The environment variable of vLLM's CPU backend that sets which CPU cores the OpenMPOpenMPAn API for shared-memory parallel programming in C, C++, and Fortran, made of compiler directives, library routines, and environment variables. A program marks loops or regions to run in parallel, and the OpenMP runtime splits them across a team of threads, which can be bound to specific CPU cores.OpenMPGlossary threads of each worker run on. It takes CPU lists such as 0-31, one per worker, separated by vertical bars, so 0-31|32-63 binds two workers to 32 cores each. The value auto binds each worker to the cores of one NUMA node, and nobind turns the binding off. See vLLM CPU installation.