Skip to main content

vLLM CPU Base (AVX2)

amd64 vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary vllm-openai CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary image built with VLLM_CPU_AVX2=true for hosts that have AVX2AVX2: Advanced Vector Extensions 2A set of x86 SIMD instructions that operate on 256-bit registers, introduced by Intel with the Haswell generation in 2013 and supported by AMD since 2015. It extends most integer vector instructions of the earlier AVX set to 256 bits. Nearly all current x86 CPUs support it, and AVX-512 is its wider successor.WikipediaGlossary (Advanced Vector Extensions 2) but not AVX-512AVX-512: Advanced Vector Extensions 512A set of x86 SIMD instructions that operate on 512-bit registers, so that one instruction processes, for example, 16 single-precision floating-point numbers at once. Intel introduced it in its server CPUs with the Skylake generation, and AMD supports it from the Zen 4 generation. It is made of several subsets, such as the foundation, avx512f, and avx512_bf16 for bfloat16 arithmetic, which Linux lists as flags in /proc/cpuinfo.WikipediaGlossary (Advanced Vector Extensions 512).

Hub vllm/vllm-openai-cpu amd64 targets AVX-512; benchmark-vllm-cpu-avx2 uses this base.

Published as ghcr.io/sparecores/vllm-cpu-base-avx2:main (amd64 only). The folder's prepare.sh hook clones the pinned source, sizes compile parallelism on physical RAMRAM: Random-Access MemoryThe main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off.WikipediaGlossary, and patches docker/Dockerfile.cpu; the build framework builds it on linux/amd64. Version pin: VLLM_VERSION.

CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary exports layer cache to GitHub Actions cache (type=gha, scope vllm-cpu-base-avx2-amd64-v{VLLM_VERSION}) and GHCRGHCR: GitHub Container RegistryGitHub's registry of container images, part of GitHub Packages, where images are published as ghcr.io/<owner>/<image>:<tag>. Each image gets a package page on GitHub that shows its description, a docker pull command, and the README of the repository it is linked to.GitHub docsGlossary (GitHub Container Registry), as buildcache-amd64. The first successful build populates both; reruns on the same runner class should restore GHA cache quickly. If buildcache-amd64 does not exist yet, registry import is skipped and the build still runs (registry cache docs).