CI and Build Internals
resource-tracker
Benchmark images copy the static
resource-tracker binary
from ghcr.io/sparecores/resource-tracker. The
release version is defined in
images/resource-tracker/RESOURCE_TRACKER_VERSION.
CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary publishes per-arch tags (main-amd64, main-arm64) during the build matrix;
the per-level merge step then creates the main multi-arch manifest.
Benchmark image builds use
RESOURCE_TRACKER_IMAGE=ghcr.io/sparecores/resource-tracker:main-<arch> so
they can resolve the binary without waiting for the merged tag.
To rebuild resource-tracker and all its consumer images without registry cache, use one of the following:
- For push builds, set the
FORCE_REBUILD_RESOURCE_TRACKERenv var in.github/workflows/push.ymlto"true", and set it back to"false"once the consumers have been rebuilt. - For a manual build, tick
force_rebuild_resource_trackerunder Actions → Build and publish Spare Cores images → Run workflow.
CI Build Tracking
Every image build job runs resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary in standalone mode (no
wrapped command, no --pid) while docker buildx build runs in parallel.
The docker buildx command only drives buildkitd over the APIAPI: Application Programming InterfaceA defined set of requests and responses through which one program uses another, such as a web service's HTTP endpoints or a library's functions.WikipediaGlossary, so compiles
do not appear in a shell-wrapper process tree; without a tracked PID,
resource-tracker collects system-wide CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary, memory, disk, and network (see
Usage.md:
"Leave unset [pid] to collect system-wide metrics only"). Metrics are sent
to Sentinel when the repo secret
SENTINEL_API_TOKEN is set (Settings → Secrets and variables → Actions →
Secrets). Without the token, tracking runs locally only and Sentinel upload is
skipped.
Job metadata (mirrors sc-inspector benchmark runs):
| Variable | CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary default |
|---|---|
TRACKER_PROJECT_NAME | sc-images |
TRACKER_JOB_NAME | matrix entry name, e.g. benchmark-vllm-gpu (arm64) |
TRACKER_TASK_NAME | image folder, e.g. benchmark-vllm-gpu |
TRACKER_STAGE_NAME | workflow name |
TRACKER_EXTERNAL_RUN_ID | github.run_id-github.run_attempt |
TRACKER_CONTAINER_IMAGE | ghcr.io/sparecores/<folder>:main-<arch> |
TRACKER_ORCHESTRATOR | github-actions |
TRACKER_ENV | ci |
TRACKER_QUIET | true (no metric spam in build logs) |
Scripts:
install-resource-tracker.sh,
run-with-resource-tracker.sh,
build-push-image.sh.
CI Runners
Push builds use GitHub-hosted runners (ubuntu-latest for amd64,
ubuntu-24.04-arm for arm64). The exception is
vllm-cpu-base-avx2, which always builds on
[self-hosted, Linux, X64, workhorse]
(build-level.yml).
Manual Run workflow checkboxes (default: off):
| Input | When enabled |
|---|---|
self_hosted_x64 | amd64 → [self-hosted, Linux, X64] |
self_hosted_arm64 | arm64 → [self-hosted, Linux, Arm64] |
Images with a ZRAM metadata file run
.github/scripts/setup-zram.sh before the
build (PERCENT=125 by default → compressed swap ≈ 1.25× RAMRAM: Random-Access MemoryThe main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off.WikipediaGlossary). Currently
enabled for vllm-cpu-base-avx2
(source-compiled CPU image) and benchmark-ffmpeg.
On GitHub-hosted Azure kernels, zramzramA Linux kernel module that creates compressed block devices in memory. It is most often used as swap, so swapped-out pages are compressed in RAM instead of written to disk, trading CPU time for memory.Linux kernel docsGlossary is in
linux-modules-extra-$(uname -r)
(Launchpad #1762756);
the script installs that package when needed. If the extra-modules deb is
missing from apt (kernel/image drift), it keeps the default /swapfile.
Tune the default percent via workflow env ZRAM_PERCENT, or override per
image with a numeric ZRAM value.
vllm-cpu-base-avx2 compile parallelism is
computed by
.github/scripts/compute-build-parallelism.sh
(RAM-linear on self-hosted, conservative on GitHub-hosted). GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary images use the
multi-arch Hub vllm/vllm-openai
base, so there is no in-repo CUDA source build.
vLLM Source Builds
The following subsections apply only to source-built vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary images, currently
vllm-cpu-base-avx2 on the self-hosted
workhorse runner.
sccache
Images with a SCCACHE metadata file compile C++/CUDA through
sccache with an S3 backend (bucket in
us-west-2, 90-day lifecycle).
Repo configuration (set by Pulumi after pulumi up):
| Name | Kind | Purpose |
|---|---|---|
SCCACHE_BUCKET | Actions variable | S3 bucket name |
SCCACHE_REGION | Actions variable | Bucket region (us-west-2) |
SC_IMAGES_GHA_ROLE_ARN | Actions secret | OIDC role for read/write on the bucket |
Each build uses prefix {folder}/{arch}/ (e.g. vllm-cpu-base-avx2/amd64/)
via SCCACHE_S3_KEY_PREFIX. Docker layer cache is exported to GHCR
(buildcache-<arch>) and GitHub Actions cache (type=gha, scope
{folder}-{arch} or {folder}-{arch}-v{VLLM_VERSION} for source-built vLLM
CPU base). The sccache tool caches compiler objects inside the vLLM source
compile steps.
Coverage when enabled:
| Stage | Mechanism |
|---|---|
rust-build | RUSTC_WRAPPER=/usr/bin/sccache for cargo; CC/CXX absolute paths for build-script C compiles (openssl-sys, etc.) |
csrc-build / vllm-build wheel | CMake compiler launchers via setup.py (CMAKE_*_COMPILER_LAUNCHER=sccache); RUSTC_WRAPPER unset so C++ is not double-wrapped |
Version 0.15.0 of sccache is installed to /usr/bin/sccache. The
SCCACHE_IGNORE_SERVER_IO_ERROR=1 setting falls back to local compiles if the
cache server is unreachable.
Currently enabled for vllm-cpu-base-avx2.
Sharing the Layer Cache Between GitHub- and Self-Hosted Runners
A slow image can be pre-warmed on a self-hosted runner and then finished quickly on a GitHub-hosted one, but only if both produce identical BuildKit cache keys. Two things break that by default, and both are handled for source-built images:
- Floating base tags and per-runner build args.
ARG max_jobsandENV CARGO_BUILD_JOBSare baked to the 64 GiB reference values andFROM ubuntu:22.04is pinned by digest, so the Dockerfile is byte-identical everywhere (.github/scripts/tune-vllm-cpu-dockerfile.shprints itssha256). Actual parallelism is injected at build time through thevllm_parallelismBuildKit secret, whose content is excluded from the cache key. - Build context ownership. BuildKit hashes local contexts with a V1 tarsum
covering name, mode, uid, gid, size, and content. It excludes
mtimebut not uid/gid. GitHub-hosted runners check out asrunner, self-hosted ones asubuntu, so identical sources would hash differently and everyCOPY(plus the entire graph below it) would miss..github/scripts/normalize-context-ownership.shchowns the context to a canonical0:0before the build;build-level.ymlreclaims workspace ownership on persistent runners beforeactions/checkoutcleans it.
Changing any of the above invalidates the published buildcache-<arch>, so
re-warm once on a self-hosted runner before relying on it from GitHub-hosted
runners.
Layer Compression
Published images use zstd layer compression (oci-mediatypes=true,
force-compression=true) so pulls on sc-inspector VMsVM: virtual machineA computer emulated in software on a physical host, with its own OS and a share of the host's CPUs, memory, and storage. Most cloud servers are VMs.WikipediaGlossary decompress faster than
gzipgzipA lossless file compressor of the GNU Project, first released in 1992, and its file format, specified in RFC 1952. It uses the DEFLATE algorithm, which combines LZ77 matching and Huffman coding, and is the most widely supported compression format on Unix-like systems and the web. Its levels range from 1 (fastest) to 9 (smallest output).GNU gzipGlossary (Docker's default). Requires Docker Engine ≥ 23.0 on pull clients.
Implemented in build-push-image.sh;
override with BUILD_COMPRESSION=gzip to revert to the legacy format.
CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary push compression runs inside BuildKit's Go zstd encoder
(moby/buildkit#2345); there is
no zstd -T equivalent exposed on docker buildx build. It uses limited
block-level concurrency (tied to GOMAXPROCS in the buildkitd container),
not libzstd's multi-threaded mode.