Design history
2026-11-05: Family manual
README.md became the manual of the vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary benchmarks, with
docs/workloads.md, docs/usage.md, and docs/contributors.md, which
replace the image docs' copies of the shared methodology. The notes below are
from the earlier README.md, the maintainers' notes on the harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary. They are
kept as written, for the history of the design. Some of their numbers predate
the current harness (tuning_version 9); the manual describes the current
rules.
Autoconfig (budget-first)
When BENCHMARK_VLLM_AUTOCONFIG=1 (default), the harness derives GuideLLMGuideLLMAn open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports.GuideLLM on GitHubGlossary
load and vLLM server knobs from vCPUvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary count and RAMRAM: Random-Access MemoryThe main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off.WikipediaGlossary, then fits them into
the 2h overall time budget (see OVERALL_TIMEOUT_SEC in benchmark.py).
Load scales sub-linearly with vCPU (open-ended — no hard 500/512 caps),
but wall time per run is capped so a 896 vCPU box does not run for days.
| Knob | Scales with vCPU | Bounded by |
|---|---|---|
max_concurrency | sub-linear (~vCPU^0.55–0.65), floored by DP replicas on CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary, at 64 on GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary | GUIDELLM__MAX_CONCURRENCY env override |
max_seconds per strategy | workload ctx (2048 vs 4096) + budget / sweep | rag gets ≥2× chat floor before sweep shrink; only per-stage limit in autoconfig |
--rampup | not vCPU — 15% of max_seconds, cap 8s on CPU, 2–10s on GPU | CPU: must leave ≥25s of the throughput stage at peak concurrency |
sweep_size | log(vCPU) capped at 4 on CPU, then shrunk to fit per_run_budget | per_run_budget_sec (45–240s); fewer steps for rag/long ctx |
| CPU TP / DP | NUMA + model size (small models get more DP replicas) | MAX_CPU_DP, RAM for weight copies, BENCHMARK_VLLM_CPU_TP / _DP |
max_num_seqs, KV fraction | sub-linear + model RAM | available memory |
| dtype (CPU) | CPU flags, not model or arch | bfloat16bfloat16A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively.WikipediaGlossary with bf16 / avx512_bf16, else float32float32A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory.WikipediaGlossary |
Example CPU load at different sizes (8 model×workload runs, 2h budget, SmolLM2-135M):
| vCPU (NUMA) | DP | max_conc | sweep | rampup | sec/strategy | measure window |
|---|---|---|---|---|---|---|
| 1 (1) | 1 | 6 | 2 | 8s | 120 | 112s |
| 96 (2) | 6 | 116 | 4 | 8s | 60 | 52s |
| 192 (4) | 12 | 182 | 4 | 8s | 60 | 52s |
| 896 (8) | 32 | 497 | 4 | 8s | 60 | 52s |
measure window is --max-seconds minus throughput-stage --rampup.
GuideLLM counts rampup inside the stage budget; vCPU-scaled rampup (old
v/4, 24s at 96 cores / 30s at ≥128 with ~40s stages) left almost no time at
peak concurrency — that was the sharp drop after 96 CPUs. Extra DP keeps small
models at ~16 threads/rank instead of one giant OpenMPOpenMPAn API for shared-memory parallel programming in C, C++, and Fortran, made of compiler directives, library routines, and environment variables. A program marks loops or regions to run in parallel, and the OpenMP runtime splits them across a team of threads, which can be bound to specific CPU cores.OpenMPGlossary team.
Autoconfig runs guidellm run --profile kind=sweep,... with a
--constraint kind=max_duration (no max-requests by default). Sweep step
count and throughput ramp-up are profile attributes (sweep_size,
rampup_duration) — GuideLLM 0.7+ CLI shape. A 192-vCPU metal run can
schedule ~182 concurrent streams; a 1-vCPU box caps at 6. Subprocess wall time
adds warmup, rampup, one extra stage for in-flight drain, and margin so heavy
rag sweeps are not killed early.
Per workload (chat / rag / long), autoconfig restarts vllm serve with that
workload's max_model_len (2048 / 4096 / 8192) so small-RAM hosts do not
reserve KV for unused long-context headroom. Before starting a workload,
workload_kv_fits() skips combos that would KV-OOM on CPU (weights-only
model_fits is not enough for long ctx). Budget planning includes the extra
startup time. Host vCPU/RAM come from the server table when querying the DB.
Tensor / data parallelism
CPU (multi-NUMA): vLLM's VLLM_CPU_OMP_THREADS_BIND=auto drops SMT on x86
(one logical CPU per physical core) and, with a single rank, binds only to NUMA
node 0. The harness therefore writes an explicit bind string with one
logical CPU per physical core, partitioned NUMA-locally across TP×DP ranks.
Dockerfile ENV …=auto means "let the harness decide".
Divides --gpu-memory-utilization--gpu-memory-utilizationThe vLLM option that sets the fraction of device memory, from 0 to 1, that a vLLM worker may use for the model weights, intermediate results, and key-value cache. On GPUs, it applies to the memory of each GPU; on the CPU backend of vLLM, each worker applies it to the total memory of its NUMA node, despite the option's name. The memory left after the weights and intermediate results goes to the key-value cache, so a higher value lets the server hold more concurrent sequences.vLLM engine argumentsGlossary by the number of ranks sharing a NUMA node.
On CPU that flag is a fraction of the rank's own memory node, and every
worker reserves it independently, so an undivided 0.50 with DP=2 asks for 100%
of the node and kills a rank with Available memory on node 0 … is less than requested memory for kv. Their sum stays under CPU_NODE_MEMORY_BUDGET (with
an extra multi-rank derate) so sequential KV allocation still fits; if that
leaves too little KV cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary the workload is skipped by workload_kv_fits()
rather than crashed.
Resource sizing no longer relies on parameter-count rules of thumb when the
Hub is reachable. --plan-only resolves:
- exact selected weight-file bytes from Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary file metadata (using the safetensors/bin index to avoid counting duplicate formats);
- layers, attention/KV heads, hidden size, and dtype from
config.json; - KV bytes/token as
2 × layers × KV-heads-per-TP-rank × head_dim × dtype_bytes; - actual per-NUMA-node memory from sysfs and per-device VRAM from CUDA.
Every CPU layout is checked on the node where each worker's first bound CPU
causes vLLM to allocate. Thus nobind correctly places every worker on one
node and is rejected when that node cannot hold all weight copies plus minimum
KV cache, even if aggregate host RAM is ample.
This preflight is deliberately conservative but is not a replacement for
vLLM initialization: GPU non-KV memory (CUDA graphs, NCCL, peak activations)
is profiled only after the worker starts. The harness therefore also extracts
vLLM's runtime Available KV cache memory, KV tokentokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary capacity, and suggested
--kv-cache-memory-bytes--kv-cache-memory-bytesThe vLLM option that sets the size of the key-value cache of each worker in bytes. Without it, vLLM derives the cache size from --gpu-memory-utilization; this option gives finer control over how much memory the cache takes.vLLM engine argumentsGlossary values into each JSONLJSON LinesA text format that stores one JSON value, usually an object, per line, so a file can be written and read one record at a time. It suits logs and streamed results, and line-based tools such as grep can process it. Files in this format usually end in .jsonl.JSON LinesGlossary row's runtime_memory.
Context requirements are rounded to vLLM cache blocks (CPU 128, GPU 16), and
the 70B bitsandbytesbitsandbytesAn open-source Python library that quantizes the weights of PyTorch models to 8 or 4 bits, so that large models need less GPU memory. Hugging Face Transformers and vLLM can load models quantized with it, and its 4-bit format is also used to fine-tune large models on a single GPU.bitsandbytes on GitHubGlossary entry is planned with the same pipeline parallelismpipeline parallelismA way to split a model across several devices by assigning consecutive groups of its layers to each, so that each device holds only part of the weights. The output of one stage passes to the next, and the stages work on different requests or micro-batches at the same time to keep all devices busy. It needs less communication between devices than tensor parallelism.Hugging Face docsGlossary used
by the server.
On a 2-GPU host, default ladder TP:
| Model | Attention heads | TP | GPUs used |
|---|---|---|---|
| SmolLM2-135M | 9 | 1 | GPU 0 only (9 % 2 ≠ 0) |
| Qwen2.5-0.5B | 14 | 2 | both |
| Gemma-2-2B | 8 | 2 | both |
| Llama-3.1-8B | 32 | 2 | both |
| Phi-4 | 40 | 2 | both |
| Llama-3.3-70B bnb-4bit | — | pipeline-parallel-size 2 | both |
SmolLM on 2×GPU is still a real GPU run (mode=gpu in JSONL); nvidia-smi pmon showing ~95% SMSM: Streaming MultiprocessorThe basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number.CUDA programming guideGlossary (Streaming Multiprocessor) use on one GPU and idle on the
other is expected for TP=1.
Images and version bumps
Inspector tries GPU → Hub CPU → AVX2 CPU; first successful probe runs the full benchmark.
Bump versions: edit VLLM_VERSION / GUIDELLM_VERSION, push to main; CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary
rebuilds vllm-cpu-base-avx2 (when pinned) and benchmark-vllm-*. For GPU,
confirm vllm/vllm-openai:v{VLLM_VERSION} is multi-arch on
Docker Hub before merging.