vLLM CPU
This benchmark measures how fast AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary CPUsCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary with AVX-512AVX-512: Advanced Vector Extensions 512A set of x86 SIMD instructions that operate on 512-bit registers, so that one instruction processes, for example, 16 single-precision floating-point numbers at once. Intel introduced it in its server CPUs with the Skylake generation, and AMD supports it from the Zen 4 generation. It is made of several subsets, such as the foundation, avx512f, and avx512_bf16 for bfloat16 arithmetic, which Linux lists as flags in /proc/cpuinfo.WikipediaGlossary (Advanced Vector
Extensions 512) and Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossary CPUs serve LLMsLLM: Large Language ModelA neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code.WikipediaGlossary to many concurrent requests. It is
the CPU image of the vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary benchmarks: it starts each default model with vLLM
(vllm serve) on the CPU and sends synthetic requests to it with GuideLLMGuideLLMAn open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports.GuideLLM on GitHubGlossary,
from one request at a time up to peak load.
The vLLM benchmarks manual describes the methodology, settings, and output that all vLLM benchmarks share, and this page covers only what this image does differently.
Like the other vLLM benchmarks, this benchmark reports the latency and throughput metrics of each model under load; see the manual's Results.
Purpose
This image runs the vLLM benchmarks on CPUs, with the
official vLLM CPU server images
from Docker HubDocker HubDocker's public registry of container images, where projects publish their images under a name and tag, such as ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry.Docker docsGlossary. Production tries it on every server type where
benchmark-vllm-gpu does not run.
On AMD64 CPUs without AVX-512, which the official image does not support,
benchmark-vllm-cpu-avx2 runs the
same benchmark instead. See the manual's
Purpose for how the vLLM benchmarks
relate to the LLM inference speedLLM inference speedHow fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generation, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.Glossary that Spare Cores Navigator publishes.
Limitations
The results of this image share the limitations of the vLLM benchmarks on CPUs. Note the following limitation of this image in addition:
- AVX-512 on AMD64: On AMD64 CPUs, this
benchmark needs AVX-512. Without it, it exits with code 2, and its error
message points to
benchmark-vllm-cpu-avx2. Arm64 CPUs skip this check.
Usage
Run the image via Docker with the container settings of production, with
<limit> set as the manual's
Production Setup
describes:
docker run --rm --privileged --network host --shm-size=4g --memory=<limit> --memory-swap=<limit> --memory-swappiness=0 -e HF_TOKEN ghcr.io/sparecores/benchmark-vllm-cpu:main > results.jsonl 2> results.log
A minimal run needs only --shm-size=4g and -e HF_TOKEN; the other
options of the first command only reproduce production:
docker run --rm --shm-size=4g -e HF_TOKEN ghcr.io/sparecores/benchmark-vllm-cpu:main > results.jsonl 2> results.log
See the manual's Usage for the Hugging Face tokenaccess tokenA secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted.Hugging Face Hub docsGlossary, the model cache, the 2-hour time budget, and the settings that this image takes.
Requirements and Versions
The image is built on the official vLLM CPU server images from Docker HubDocker HubDocker's public registry of container images, where projects publish their images under a name and tag, such as ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry.Docker docsGlossary,
vllm/vllm-openai-cpu:v0.27.1-x86_64 for AMD64 and
vllm/vllm-openai-cpu:v0.27.1-arm64 for Arm64, and its :main tag has
variants for both.
At startup on AMD64, this benchmark reads the CPU flags from /proc/cpuinfo.
The vLLM CPU image for AMD64 needs AVX-512, so without the avx512f flag,
this benchmark logs an error that names
ghcr.io/sparecores/benchmark-vllm-cpu-avx2:main and exits with code 2.
Setting BENCHMARK_VLLM_ALLOW_AVX2_ONLY to 1, true, or yes skips this
check; benchmark-vllm-cpu-avx2 sets it, and this image does not.
Arm64 CPUs skip the check, and this benchmark does not check the Arm64 CPU
generation either. On Arm64 CPUs without the bf16 flag, it runs the models
in float32float32A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory.WikipediaGlossary; see Data Type.
Results
The JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary lines in stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary have the fields that the manual's
Results and
Output Fields describe for
CPU runs. In this image, arch is amd64 or arm64, avx512 is true on
AMD64 and null on Arm64, and avx2_only_image is false.
The process exits with the codes of the manual's Exit Codes, and with code 2 on an AMD64 CPU without AVX-512.
Workloads
This benchmark runs the chat and rag workloads of the vLLM benchmarks on
the five default models that run on CPUs; see the manual's
Workloads.
References
The terms of the vLLM benchmarks, such as --max-model-len--max-model-lenThe vLLM option that sets the context length of a model, the maximum number of tokens, prompt and output together, in one sequence. The key-value cache must have room for at least one sequence of this length, so a shorter context length needs less memory. Without the option, vLLM uses the length from the model's configuration.vLLM engine argumentsGlossary, are defined in
their references.
Common terms, such as GuideLLM, NUMANUMA: Non-Uniform Memory AccessA memory design where each CPU socket or chiplet has its own local memory, which its cores reach faster than the memory of the others. Software that keeps its threads and their memory on the same NUMA node avoids the slower remote accesses.Linux kernel docsGlossary (Non-Uniform Memory Access), and
AVX-512, are defined in the glossary
of the Spare Cores docs. To change or rebuild this image, see the
contributor guide of the vLLM
benchmarks.