Skip to main content

vLLM GPU

This benchmark measures how fast NVIDIA GPUs serve LLMsLLM: Large Language ModelA neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code.WikipediaGlossary to many concurrent requests. It is the GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary image of the vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary benchmarks: it starts each default model with vLLM (vllm serve), from the smallest to the largest, and sends synthetic requests to it with GuideLLMGuideLLMAn open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports.GuideLLM on GitHubGlossary, from one request at a time up to peak load.

The vLLM benchmarks manual describes the methodology, settings, and output that all vLLM benchmarks share, and this page covers only what this image does differently.

Like the other vLLM benchmarks, this benchmark reports the latency and throughput metrics of each model under load; see the manual's Results.

Purpose​

This image runs the vLLM benchmarks on NVIDIA GPUs, with the official vLLM server image from Docker HubDocker HubDocker's public registry of container images, where projects publish their images under a name and tag, such as ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry.Docker docsGlossary. Production tries it first on every server type, and runs the CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary images only where it does not run. See the manual's Purpose for how the vLLM benchmarks relate to the LLM inference speedLLM inference speedHow fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generation, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.Glossary that Spare Cores Navigator publishes.

Limitations​

The results of this image share the limitations of the vLLM benchmarks on GPUs. Note the following limitation of this image in addition:

  • NVIDIA GPUs Only: This benchmark needs NVIDIA GPUs with CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture) compute capabilitycompute capabilityA version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later.NVIDIAGlossary 8.0 or newer, the Ampere generation or later. If no GPU is visible or every visible GPU is older, it exits with code 1.

Usage​

Run the image via Docker with the GPUs passed and 4 GiB of shared memory:

docker run --rm --gpus all --shm-size=4g -e HF_TOKEN ghcr.io/sparecores/benchmark-vllm-gpu:main > results.jsonl

To reproduce the container settings of production, except the memory limit, run the following command:

docker run --rm --privileged --network host --gpus all --shm-size=4g -e HF_TOKEN ghcr.io/sparecores/benchmark-vllm-gpu:main > results.jsonl

See the manual's Usage for the Hugging Face tokenaccess tokenA secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted.Hugging Face Hub docsGlossary, the model cache, the 2-hour time budget, and the settings that this image takes.

GPU Run​

The host needs the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary and the NVIDIA Container ToolkitNVIDIA Container ToolkitA set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container.NVIDIA docsGlossary for --gpus. The image is built on vllm/vllm-openai:v0.27.1 from Docker HubDocker HubDocker's public registry of container images, where projects publish their images under a name and tag, such as ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry.Docker docsGlossary, and its :main tag has variants for AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary and Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossary.

At startup, this benchmark reads the CUDA compute capabilitycompute capabilityA version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later.NVIDIAGlossary of each visible GPU. It needs compute capability 8.0 or newer, the Ampere generation or later, such as the NVIDIA A10GNVIDIA A10GA data center GPU of NVIDIA's Ampere generation with 24 GB of GDDR6 memory and compute capability 8.6, a variant of the NVIDIA A10 made for AWS. It powers the EC2 G5 instances, which are used for machine learning inference and graphics workloads.AWSGlossary or L40SNVIDIA L40SA data center GPU of NVIDIA's Ada Lovelace generation with 48 GB of GDDR6 memory and compute capability 8.9. It is built for AI inference and training as well as graphics and video workloads.NVIDIAGlossary, but not the older T4GNVIDIA T4GA data center GPU of NVIDIA's Turing generation with 16 GB of GDDR6 memory and compute capability 7.5, a variant of the NVIDIA T4 made for AWS. It powers the EC2 G5g instances, which pair it with Arm-based AWS Graviton2 CPUs.AWSGlossary. If every visible GPU is below 8.0, it logs an error and exits with code 1. A GPU whose compute capability cannot be read is allowed.

If no GPU is visible, for example without --gpus, the check passes, but no model fits the GPU memory check. This benchmark then logs each model as skipped for insufficient memory and exits with code 1.

Results​

The JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary lines in stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary have the fields that the manual's Results and Output Fields describe for GPU runs.

Workloads​

This benchmark runs all three workloads of the vLLM benchmarks, chat, rag, and long, on the six default models; see the manual's Workloads. The Load Settings give the settings of production runs on GPU server types.

References​

The terms of the vLLM benchmarks, such as --max-model-len--max-model-lenThe vLLM option that sets the context length of a model, the maximum number of tokens, prompt and output together, in one sequence. The key-value cache must have room for at least one sequence of this length, so a shorter context length needs less memory. Without the option, vLLM uses the length from the model's configuration.vLLM engine argumentsGlossary, are defined in their references. Common terms, such as GuideLLM, CUDA, and tensor parallelismtensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary, are defined in the glossary of the Spare Cores docs. To change or rebuild this image, see the contributor guide of the vLLM benchmarks.