Contributor Guide
Image Layout
The three vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary images differ only in their Dockerfile and base image. Each
Dockerfile installs psutilpsutilA cross-platform Python library for retrieving information on running processes and system utilization, such as CPU, memory, disks, and network. It can count both logical CPUs and physical cores.psutil docsGlossary, and guidellm[recommended] at the pinned
GuideLLMGuideLLMAn open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports.GuideLLM on GitHubGlossary version, with pip, and copies the following files:
- the
resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary binary, from the image set byRESOURCE_TRACKER_IMAGE benchmark.py,VLLM_VERSION, andGUIDELLM_VERSIONfromvllm-common/, to/
The images share the harness and the version files, so
their build context is the root of the
sc-images repository, as each
image's CONTEXT file sets. The container entrypointcontainer entrypointThe executable that a container image runs when a container starts, set by the ENTRYPOINT instruction of its Dockerfile. The image's default command (CMD), or the arguments given after the image name in docker run, are passed to it as arguments, so an entrypoint can wrap any command, for example to set up the environment or record resource usage first. The docker run --entrypoint option replaces it.Dockerfile referenceGlossary of each image is
resource-tracker -- python3 /benchmark.py.
| Image | Base image | Architectures |
|---|---|---|
benchmark-vllm-gpu | vllm/vllm-openai:v<VLLM_VERSION> from Docker HubDocker HubDocker's public registry of container images, where projects publish their images under a name and tag, such as ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry.Docker docsGlossary | amd64, arm64 |
benchmark-vllm-cpu | vllm/vllm-openai-cpu:v<VLLM_VERSION>-x86_64 for amd64 and vllm/vllm-openai-cpu:v<VLLM_VERSION>-arm64 for arm64 from Docker Hub, selected by the TARGETARCH build argument | amd64, arm64 |
benchmark-vllm-cpu-avx2 | ghcr.io/sparecores/vllm-cpu-base-avx2:main-amd64, vLLM built from source for AVX2 by vllm-cpu-base-avx2 | amd64 |
The images set the following environment variables. /models is a volume, and
USER, HOME, and TORCHINDUCTOR_CACHE_DIR are for runs as a non-root user
with --user; no Dockerfile has a USER instruction:
| Variable | benchmark-vllm-gpu | benchmark-vllm-cpu | benchmark-vllm-cpu-avx2 |
|---|---|---|---|
BENCHMARK_VLLM_MODE | gpu | cpu | cpu |
BENCHMARK_VLLM_ALLOW_AVX2_ONLY | Unset | Unset | true |
VLLM_CPU_OMP_THREADS_BINDVLLM_CPU_OMP_THREADS_BINDThe environment variable of vLLM's CPU backend that sets which CPU cores the OpenMP threads of each worker run on. It takes CPU lists such as 0-31, one per worker, separated by vertical bars, so 0-31|32-63 binds two workers to 32 cores each. The value auto binds each worker to the cores of one NUMA node, and nobind turns the binding off.vLLM CPU installationGlossary | Unset | auto | auto |
TRACKER_QUIET | true | true | true |
HF_HOME, HUGGINGFACE_HUB_CACHE | /models | /models | /models |
USER | benchmark | benchmark | benchmark |
HOME | /tmp | /tmp | /tmp |
TORCHINDUCTOR_CACHE_DIR | /tmp/torch_inductor | /tmp/torch_inductor | /tmp/torch_inductor |
Build Arguments
| Argument | Meaning | Default |
|---|---|---|
VLLM_VERSION | benchmark-vllm-gpu and benchmark-vllm-cpu: tag of the base image, without the v prefix and the architecture suffix | 0.27.1 |
VLLM_CPU_BASE_AMD64 | benchmark-vllm-cpu-avx2: the AVX2 base image | ghcr.io/sparecores/vllm-cpu-base-avx2:main-amd64 |
GUIDELLM_VERSION | GuideLLM version installed with pip | 0.7.3 |
RESOURCE_TRACKER_IMAGE | Image to copy the resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary binary from. CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary passes the image for the target architecture, ghcr.io/sparecores/resource-tracker:main-<arch>. | ghcr.io/sparecores/resource-tracker:main |
CI reads VLLM_VERSION and GUIDELLM_VERSION from the files in
vllm-common/, and builds each image for its architectures, after the
resource-tracker image it depends on. The Dockerfile defaults match the
files. The benchmark-vllm-cpu-avx2 image takes its vLLM version from its
base image, which vllm-cpu-base-avx2 builds from the same
VLLM_VERSION file.
To build an image locally, run the following command from the root of the
sc-images repository, with the image's folder name, such as
benchmark-vllm-cpu:
docker build -f images/<image>/Dockerfile -t <image> .
To check the rebuilt image, print the pinned versions:
docker run --rm <image> --version
Version Changes
The version files in vllm-common/, VLLM_VERSION and GUIDELLM_VERSION,
pin the vLLM and GuideLLM versions for all vLLM images. Before changing
VLLM_VERSION, confirm on Docker HubDocker HubDocker's public registry of container images, where projects publish their images under a name and tag, such as ubuntu:24.04. Docker pulls images from it by default when a name gives no other registry.Docker docsGlossary that the new tags exist for both
architectures, because CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary builds the images for both:
vllm/vllm-openai, for bothlinux/amd64andlinux/arm64vllm/vllm-openai-cpu, both the-x86_64and the-arm64tag
Changing either version makes new results incomparable with earlier ones, so every server type would need a rerun. We do not plan to change the pinned versions.
Tests
The unit tests of benchmark.py are in
test_benchmark.py in vllm-common/. They include
GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary cases for the data-paralleldata parallelismA way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices.Hugging Face docsGlossary override, the memory fit check of each GPU,
and the compute capabilitycompute capabilityA version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later.NVIDIAGlossary check.