vLLM CPU AVX2
This benchmark measures how fast AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary CPUsCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary with AVX2AVX2: Advanced Vector Extensions 2A set of x86 SIMD instructions that operate on 256-bit registers, introduced by Intel with the Haswell generation in 2013 and supported by AMD since 2015. It extends most integer vector instructions of the earlier AVX set to 256 bits. Nearly all current x86 CPUs support it, and AVX-512 is its wider successor.WikipediaGlossary (Advanced Vector
Extensions 2) but without AVX-512AVX-512: Advanced Vector Extensions 512A set of x86 SIMD instructions that operate on 512-bit registers, so that one instruction processes, for example, 16 single-precision floating-point numbers at once. Intel introduced it in its server CPUs with the Skylake generation, and AMD supports it from the Zen 4 generation. It is made of several subsets, such as the foundation, avx512f, and avx512_bf16 for bfloat16 arithmetic, which Linux lists as flags in /proc/cpuinfo.WikipediaGlossary (Advanced Vector Extensions 512) serve LLMsLLM: Large Language ModelA neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code.WikipediaGlossary
to many concurrent requests. It runs the vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary benchmarks on a vLLM build for
these CPUs: it starts each default model with vllm serve on the CPU and
sends synthetic requests to it with GuideLLMGuideLLMAn open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports.GuideLLM on GitHubGlossary, from one request at a time up to
peak load.
The vLLM benchmarks manual describes the methodology, settings, and output that all vLLM benchmarks share, and this page covers only what this image does differently.
Like the other vLLM benchmarks, this benchmark reports the latency and throughput metrics of each model under load; see the manual's Results. Each line also marks the AVX2 image and the data type that the model ran in; see Results.
Purpose
The benchmark-vllm-cpu image is built on
the official vLLM CPU server image
from Docker HubDocker HubDocker's public registry of container images, where projects publish their images under a name and tag, such as ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry.Docker docsGlossary, whose AMD64 build needs AVX-512. On an AMD64 CPU without
AVX-512, benchmark-vllm-cpu exits with code 2 and names this image in its
error message.
This image runs the same measurements on vLLM built from source for AVX2, so that server types with these CPUs can get vLLM results too. See the manual's Purpose for how the vLLM benchmarks relate to the LLM inference speedLLM inference speedHow fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generation, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.Glossary that Spare Cores Navigator publishes.
Limitations
The results of this image share the limitations of the vLLM benchmarks on CPUs. Note the following limitations of this image in addition:
- No CPU Feature Check: This image skips the harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary's AVX-512 check and does not check for AVX2 either. Production tries it on every AMD64 server type where the two vLLM images before it in the probe order fail their probe, without checking the CPU flags first. See also Production Setup.
- Data Type and Memory: On CPUs without AVX-512, the models run in float32float32A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory.WikipediaGlossary by default. The memory check counts the weights of a bfloat16bfloat16A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively.WikipediaGlossary model at twice their size, because they run in float32, so a host can fit fewer models than a host with the same memory and bfloat16 support.
- Separate vLLM Build: This image runs vLLM
built from source for AVX2, not the official Docker HubDocker HubDocker's public registry of container images, where projects publish their images under a name and tag, such as
ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry.Docker docsGlossary image thatbenchmark-vllm-cpuruns. Both images use the same vLLM version.
Usage
Run the image via Docker with the container settings of production, with
<limit> set as the manual's
Production Setup
describes:
docker run --rm --privileged --network host --shm-size=4g --memory=<limit> --memory-swap=<limit> --memory-swappiness=0 -e HF_TOKEN ghcr.io/sparecores/benchmark-vllm-cpu-avx2:main > results.jsonl 2> results.log
A minimal run needs only --shm-size=4g and -e HF_TOKEN; the other
options of the first command only reproduce production:
docker run --rm --shm-size=4g -e HF_TOKEN ghcr.io/sparecores/benchmark-vllm-cpu-avx2:main > results.jsonl 2> results.log
See the manual's Usage for the Hugging Face tokenaccess tokenA secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted.Hugging Face Hub docsGlossary, the model cache, and the 2-hour time budget.
Requirements and Versions
This image is built on
vllm-cpu-base-avx2, which builds vLLM
from source with the VLLM_CPU_AVX2=true build option. This image and its
base are published for AMD64 only. This image and benchmark-vllm-cpu take
the vLLM version from the same pin, 0.27.1.
The Dockerfile sets BENCHMARK_VLLM_ALLOW_AVX2_ONLY=true,
which skips the harness's AVX-512 check on AMD64.
This image does not check for AVX2 itself. To check a host before a
run, look for the avx2 flag in /proc/cpuinfo; the following command prints
avx2 only when the CPU has it:
grep -o -m 1 avx2 /proc/cpuinfo
At startup, the log in stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary lists the CPU flags that the harness found, such
as cpu_flag_avx2=yes, and cpu_bf16=False on a CPU without bfloat16bfloat16A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively.WikipediaGlossary support.
On a CPU without AVX-512, the log has no cpu_flag_avx512f line.
Settings
This image takes the
options and environment variables
of the vLLM benchmarks. The only default that differs from
benchmark-vllm-cpu is BENCHMARK_VLLM_ALLOW_AVX2_ONLY, which this image sets
to true.
Results
The JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary lines in stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary have the fields that the manual's
Results and
Output Fields describe for
CPU runs. The following fields read differently in this image:
avx2_only_imageistrue, because this image setsBENCHMARK_VLLM_ALLOW_AVX2_ONLY.avx512isfalseon CPUs without AVX-512.modeiscpu, andarchisamd64.tuningholds the data type indtype, which isfloat32by default on CPUs without AVX-512.
The process exits with the codes of the manual's
Exit Codes. With its default
settings, this image does not exit with code 2, the code of
benchmark-vllm-cpu on CPUs without AVX-512, because it skips that check. Any
BENCHMARK_VLLM_ALLOW_AVX2_ONLY value other than 1, true, or yes, such
as -e BENCHMARK_VLLM_ALLOW_AVX2_ONLY=0, turns the check back on.
Production Setup
Production tries this image last of the three vLLM images, without checking
the CPU flags of an AMD64 server type first; see the manual's
Production Setup. To test
this image on a server type before a full run, run its
--probe-only check:
docker run --rm --shm-size=4g -e HF_TOKEN ghcr.io/sparecores/benchmark-vllm-cpu-avx2:main --probe-only
Workloads
This benchmark runs the chat and rag workloads of the vLLM benchmarks on
the five default models that run on CPUs, unchanged; see the manual's
Workloads.
Data Type
An AMD64 CPU without AVX-512 has no avx512_bf16 flag, so on such a CPU, this
image runs the models in float32float32A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory.WikipediaGlossary by default, and the
memory check doubles the
size of a bfloat16bfloat16A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively.WikipediaGlossary model's weights; see
Data Type.
Setting VLLM_CPU_DTYPE, such as -e VLLM_CPU_DTYPE=bfloat16, overrides the
data type, and the memory check then uses the data type set.
To see which models fit on a host before a full run, start this image with
the --plan-only
option. It prints the host's resources and whether each model fits in memory,
as JSON, without starting vLLM:
docker run --rm -e HF_TOKEN ghcr.io/sparecores/benchmark-vllm-cpu-avx2:main --plan-only
References
The terms of the vLLM benchmarks, such as --max-model-len--max-model-lenThe vLLM option that sets the context length of a model, the maximum number of tokens, prompt and output together, in one sequence. The key-value cache must have room for at least one sequence of this length, so a shorter context length needs less memory. Without the option, vLLM uses the length from the model's configuration.vLLM engine argumentsGlossary, are defined in
their references.
Common terms, such as AVX2, AVX-512, and float32float32A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory.WikipediaGlossary, are defined in the
glossary of the Spare Cores docs. To
change or rebuild this image, see the
contributor guide of the vLLM
benchmarks.