Skip to main content

vLLM

The vLLM benchmarks measure how fast CPUsCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary and NVIDIA GPUs serve LLMsLLM: Large Language ModelA neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code.WikipediaGlossary to many concurrent requests. Each run starts the default models with vLLM (vllm serve), from the smallest to the largest, and sends synthetic requests to each over HTTPHTTP: Hypertext Transfer ProtocolThe request-response protocol of the web, in which a client asks a server for a resource by URL and gets back a status code, headers, and a body. HTTPS is HTTP over an encrypted connection.MDNGlossary with GuideLLMGuideLLMAn open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports.GuideLLM on GitHubGlossary, from one request at a time up to peak load. Three images run the same harness, each on its own hardware.

The vLLM benchmarks write one JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary line per model, workload, load stage, metric, and statistic. The lines report the following metrics:

  • Latency: in milliseconds, each as the 50th, 95th, and 99th percentile (p50, p95, p99) and the mean:
    • TTFTTTFT: Time to First TokenThe time from sending a request to an LLM server until the first output token arrives. It includes queuing, prompt processing, and the generation of the first token, so it grows with the prompt length and the server's load, and it sets how long a user waits before a response starts.NVIDIA NIM docsGlossary (Time to First Token)
    • TPOTTPOT: Time per Output TokenThe average time an LLM server takes to generate each output token of a response after the first one, that is, the generation time after the first token divided by the number of further tokens. Together with the time to first token, it sets how long a full response takes.NVIDIA NIM docsGlossary (Time per Output Token)
    • ITLITL: Inter-Token LatencyThe time between two consecutive output tokens of a response that an LLM server streams. It shows how smoothly text appears while it is generated, and is usually reported as percentiles or the mean over all gaps between tokens.NVIDIA NIM docsGlossary (Inter-Token Latency)
    • E2ELE2EL: End-to-End Request LatencyThe time from sending a request to an LLM server until the last output token of its response arrives. It covers queuing, prompt processing, and the generation of every output token, so it is the time to first token plus the generation time of the further tokens, and it sets how long a full response takes.NVIDIA NIM docsGlossary (End-to-End Request Latency)
  • Throughput: each as a mean, in TPSTPS: Tokens per SecondThe number of tokens an LLM server processes or generates per second. Benchmarks count it in different ways, over output tokens only or over prompt and output tokens, per request or summed over all concurrent requests, so a TPS figure is compared only with one counted the same way.NVIDIA NIM docsGlossary (Tokens per Second) or RPSRPS: Requests per SecondThe number of requests a server completes per second over a measurement window. It measures a server's capacity in whole requests, independent of their size, so it is compared only between runs with the same request mix.NVIDIA NIM docsGlossary (Requests per Second):
    • Output TPS: output tokenstokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary generated
    • Total TPS: prompt and output tokens
    • RPS: requests served

Purpose​

Spare Cores Navigator publishes empirical performance measurements for over 5,000 cloud server types, among them LLM inference speed: how fast llama.cpp processes prompts and generates text for one request at a time.

The vLLM benchmarks measure LLM serving instead: the latency and throughput of a vLLM server under concurrent load, from one request at a time up to peak load. In its note on the LLM inference speedLLM inference speedHow fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generation, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.Glossary scores, the Navigator points readers who need high concurrency to the vLLM benchmarks, but the Navigator does not show their results yet.

The harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary runs in the following images, in the order that production tries them:

  • benchmark-vllm-gpu: NVIDIA GPUs with CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture) compute capabilitycompute capabilityA version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later.NVIDIAGlossary 8.0 or newer
  • benchmark-vllm-cpu: AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary CPUs with AVX-512AVX-512: Advanced Vector Extensions 512A set of x86 SIMD instructions that operate on 512-bit registers, so that one instruction processes, for example, 16 single-precision floating-point numbers at once. Intel introduced it in its server CPUs with the Skylake generation, and AMD supports it from the Zen 4 generation. It is made of several subsets, such as the foundation, avx512f, and avx512_bf16 for bfloat16 arithmetic, which Linux lists as flags in /proc/cpuinfo.WikipediaGlossary (Advanced Vector Extensions 512), and Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossary CPUs
  • benchmark-vllm-cpu-avx2: AMD64 CPUs with AVX2AVX2: Advanced Vector Extensions 2A set of x86 SIMD instructions that operate on 256-bit registers, introduced by Intel with the Haswell generation in 2013 and supported by AMD since 2015. It extends most integer vector instructions of the earlier AVX set to 256 bits. Nearly all current x86 CPUs support it, and AVX-512 is its wider successor.WikipediaGlossary (Advanced Vector Extensions 2) but without AVX-512

Each image's own page covers what only that image does: its base image, its hardware check, and its run command. This manual covers the rest, with CPU and GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary sections where the two differ.

Limitations​

These results compare LLM serving under load. Note the following limitations of all vLLM benchmarks; the image pages add the hardware limits of each image:

  • Load Follows vCPU Count: Concurrency, GuideLLM workers, and the number of load stages scale with the host's vCPUvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary count, on GPUs too: not with the GPU count, GPU model, or GPU memory. On CPUs, the vLLM batch limits scale with it as well.
  • Data Type Follows the CPU: On CPUs, models run in bfloat16bfloat16A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively.WikipediaGlossary on CPUs that support it, and in float32float32A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory.WikipediaGlossary on the others, so two server types can serve the same model in different data types.
  • vLLM Server Defaults on GPUs: On GPUs, vLLM runs with its own batch limits and data type. The vLLM server settings recorded in each line's tuning object are not passed to vllm serve.
  • One Thread per Physical Core: On CPUs, vLLM runs with one thread on each physical core, so it does not use the second SMTSMT: Simultaneous MultithreadingA CPU technique that runs two (or more) hardware threads on one physical core, sharing its execution units, e.g. Intel Hyper-Threading. It raises throughput when a thread stalls, but the threads compete for the core. Cloud vendors usually sell each thread as a vCPU.WikipediaGlossary (Simultaneous Multithreading) thread of a core. The threads of each vLLM worker stay on one NUMANUMA: Non-Uniform Memory AccessA memory design where each CPU socket or chiplet has its own local memory, which its cores reach faster than the memory of the others. Software that keeps its threads and their memory on the same NUMA node avoids the slower remote accesses.Linux kernel docsGlossary (Non-Uniform Memory Access) node.
  • Partial GPU Use: A model can run on fewer GPUs than the server has, because its tensor-paralleltensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary size must divide its attention headattention headOne of several parallel attention computations in a layer of a transformer model. Each head projects the tokens into its own smaller query, key, and value vectors and learns which earlier tokens to focus on, and the layer combines the outputs of all heads. The number of heads per layer is fixed by the model's architecture, and methods that split a layer across devices, such as tensor parallelism, usually assign whole heads to each device.Attention Is All You NeedGlossary count. On more than one GPU, the 70B model, llama-70b, uses pipeline parallelismpipeline parallelismA way to split a model across several devices by assigning consecutive groups of its layers to each, so that each device holds only part of the weights. The output of one stage passes to the next, and the stages work on different requests or micro-batches at the same time to keep all devices busy. It needs less communication between devices than tensor parallelism.Hugging Face docsGlossary instead.
  • Skipped Models and Workloads: A model or workload that does not fit in the memory it would use, of each NUMA node on CPUs or of each GPU on GPUs, is skipped. The 4-bit llama-70b model and the long workload never run on CPUs. The run still exits with code 0 as long as another model or workload wrote results. See also GPU Memory Fit.
  • One Host, Synthetic Prompts: The vLLM benchmarks measure serving latency and throughput on one host, with synthetic prompts of fixed lengths, sent by GuideLLM from the same container as the server. They do not measure the network latency between a client and the server, or the quality of the model output.

Usage​

Each image's page gives its docker run command. Every image needs the following:

  • Shared memory: --shm-size=4g, because the vLLM benchmarks fail with Docker's default 64 MiB /dev/shm.
  • Hugging Face tokenaccess tokenA secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted.Hugging Face Hub docsGlossary: export your Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary access token as HF_TOKEN in the shell, and pass it with -e HF_TOKEN.
  • GPUs, for benchmark-vllm-gpu only: --gpus all, with the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary and the NVIDIA Container ToolkitNVIDIA Container ToolkitA set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container.NVIDIA docsGlossary on the host.

Production also runs the container with privileged mode, host networking, and a memory limit. See Production Setup before comparing results with production.

The vLLM benchmarks download the models from Hugging Face while they run, and use HF_TOKEN for the downloads and the model metadata. Two default models, gemma-2b and llama-8b, are gated modelsgated modelA model on the Hugging Face Hub whose files can be downloaded only after a logged-in user accepts the conditions set by its authors, such as a license. Programs that download it then need an access token of an account that has accepted them.Hugging Face docsGlossary: they need an access token from an account that has accepted their licenses. Without one, the vLLM benchmarks skip them and run the other models.

The models are cached in /models, a Docker volume; see Hugging Face Access and Model Cache to reuse them across runs.

A run fits its models and workloads into a 2-hour time budget. Sample production runs took from 28 minutes to 1 hour 40 minutes; see Run Duration.

Settings​

Pass options after the image name, and environment variables with -e before it. The following settings are the most common; see the full option and variable reference for all of them and their defaults:

  • --models replaces the default models with Hugging Face model IDsHugging Face model IDThe name of a model repository on the Hugging Face Hub, in the form <owner>/<model>, such as Qwen/Qwen2.5-0.5B-Instruct. Programs and libraries use it to download the model's files, and it appears as the title of the model's page on the Hub.Hugging Face Hub docsGlossary or the short names of the default models, separated by spaces, such as --models qwen-0.5b <owner>/<model>.
  • BENCHMARK_VLLM_WORKLOADS selects workloads by name, such as chat.
  • BENCHMARK_VLLM_CPU_TP and BENCHMARK_VLLM_GPU_TP override the tensor-paralleltensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary size, and BENCHMARK_VLLM_CPU_DP and BENCHMARK_VLLM_GPU_DP the data-paralleldata parallelismA way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices.Hugging Face docsGlossary size.
  • VLLM_CPU_DTYPE overrides the data type on CPUs, which is bfloat16 or float32 by default, depending on the CPU.
  • VLLM_GPU_MEMORY_UTILIZATION sets the fraction of each GPU's memory that vLLM may use (default: 0.92).
  • --plan-only prints the host's resources and whether each model fits in memory, as JSON, without starting vLLM.
  • --version prints the pinned versions, vllm=0.27.1 guidellm=0.7.3, and exits.

Results​

The process prints one JSON line to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary per model, workload, GuideLLM load stage, metric, and statistic. Logs go to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary, including the load settings, each vllm serve command, and the cause when a server fails to start. The vLLM server's own log goes to a temporary file inside the container, not to stdout or stderr. The main fields of each line are the following:

  • measurement names the metric: ttft, tpot, itl, and e2el for latency, and output_throughput, total_throughput, and request_throughput for throughput.
  • percentile is p50, p95, p99, or mean for latency, and null for throughput. The score field holds the value, and unit is ms, tokens/sec, or requests/sec.
  • model (the short name, such as llama-8b), workload, and strategy identify the run, and model_id holds the Hugging Face model ID. The strategy field is the GuideLLM load stage, synchronous, throughput, or constant, and concurrency is its mean number of concurrent requests. The target_rate field is the request rate of a constant stage, and null when GuideLLM gives no rate, as in production synchronous stages.
  • mode is cpu or gpu, and arch is amd64 or arm64.
  • tensor_parallel, pipeline_parallel, and data_parallel show how the model was split across the NUMA nodes or GPUs.
  • tuning holds the load and vLLM server settings of the run, such as the data type in dtype.
  • vllm_version and guidellm_version give the pinned versions.

See Output Fields for the CPU-only and GPU-only fields. For full examples, see the production output of an AWSAWS: Amazon Web ServicesThe cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning.AWSGlossary (Amazon Web Services) c8a.12xlarge on CPUs and of an AWS g5.16xlarge with one NVIDIA A10GNVIDIA A10GA data center GPU of NVIDIA's Ampere generation with 24 GB of GDDR6 memory and compute capability 8.6, a variant of the NVIDIA A10 made for AWS. It powers the EC2 G5 instances, which are used for machine learning inference and graphics workloads.AWSGlossary GPU.

The process exits with code 0 even when it skipped models or workloads, so check which lines are present in stdout and which reasons are logged to stderr. It exits with code 1 when no model could run or no line was written; see Exit Codes for the other codes and the log messages of skipped models.

Resource usage is recorded with resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary, which streams it to the Spare Cores Sentinel platform for the Spare Cores team's own monitoring. The resource usage is not published.

Workloads​

Each model runs the following workloads of fixed synthetic prompt and output lengths. The vLLM benchmarks restart vllm serve for each workload, with a context length that fits it:

  • chat: 256 prompt tokens and 128 output tokens, with a context of 2,048 tokens.
  • rag: 1,024 prompt tokens and 256 output tokens, with a context of 4,096 tokens.
  • long: 4,096 prompt tokens and 512 output tokens, with a context of 8,192 tokens. Only GPU runs include it.

Each workload runs one GuideLLM sweepGuideLLM sweepA GuideLLM benchmark profile that measures a server across a range of loads. It first runs a synchronous stage, which sends one request at a time, and a throughput stage, which sends as many concurrent requests as allowed. It then runs the remaining stages, if any, at constant request rates between the rates of those two stages. The sweep size is the total number of stages.GuideLLM on GitHubGlossary: a synchronous stage of one request at a time, a throughput stage at the highest concurrency allowed, and, when the time budget allows, constant-rate stages between the two. On CPUs, hosts with fewer than 8 vCPUs run no constant-rate stages.

See Models for the six default models, from 135 million to 70 billion parameters, five of which run on CPUs, and Run Sequence for when a model is skipped or stops the run.

Before it starts the server for each model, the harness does the following:

  • It splits the model across the NUMA nodes and CPU cores, or across the GPUs; see CPU Parallelism and GPU Parallelism.
  • It checks that the model fits in the memory of each NUMA node or each GPU; see CPU Memory Fit and GPU Memory Fit.
  • On CPUs, it picks a data type that the CPU supports; see Data Type.
  • It derives the load from the host's vCPU count, and on CPUs also the vLLM batch limits; see Load Settings.

Design History​

The harness first ran fixed load settings on every host. It now derives the load from the host's vCPU count and fits all runs into a 2-hour budget, so that a host with hundreds of vCPUs gets more load without running for days. The fixed settings remain as Legacy Settings for comparisons with older results.

An early version also scaled the throughput stage's ramp-up with the vCPU count, which left almost no time at peak concurrency on hosts with more than 96 vCPUs. The ramp-up is now a share of the stage, capped at 8 seconds on CPUs and 10 seconds on GPUs. See the changelog for the details of both changes.

References​

The terms of the vLLM benchmarks, such as --max-model-len--max-model-lenThe vLLM option that sets the context length of a model, the maximum number of tokens, prompt and output together, in one sequence. The key-value cache must have room for at least one sequence of this length, so a shorter context length needs less memory. Without the option, vLLM uses the length from the model's configuration.vLLM engine argumentsGlossary, are defined in the References. Common terms, such as GuideLLM, TTFT, and tensor parallelismtensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary, are defined in the glossary of the Spare Cores docs. To change or rebuild the vLLM images, see the contributor guide.