Skip to main content

CI and Build Internals

resource-tracker​

Benchmark images copy the static resource-tracker binary from ghcr.io/sparecores/resource-tracker. The release version is defined in images/resource-tracker/RESOURCE_TRACKER_VERSION.

CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary publishes per-arch tags (main-amd64, main-arm64) during the build matrix; the per-level merge step then creates the main multi-arch manifest. Benchmark image builds use RESOURCE_TRACKER_IMAGE=ghcr.io/sparecores/resource-tracker:main-<arch> so they can resolve the binary without waiting for the merged tag.

To rebuild resource-tracker and all its consumer images without registry cache, use one of the following:

  • For push builds, set the FORCE_REBUILD_RESOURCE_TRACKER env var in .github/workflows/push.yml to "true", and set it back to "false" once the consumers have been rebuilt.
  • For a manual build, tick force_rebuild_resource_tracker under Actions → Build and publish Spare Cores images → Run workflow.

CI Build Tracking​

Every image build job runs resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary in standalone mode (no wrapped command, no --pid) while docker buildx build runs in parallel. The docker buildx command only drives buildkitd over the APIAPI: Application Programming InterfaceA defined set of requests and responses through which one program uses another, such as a web service's HTTP endpoints or a library's functions.WikipediaGlossary, so compiles do not appear in a shell-wrapper process tree; without a tracked PID, resource-tracker collects system-wide CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary, memory, disk, and network (see Usage.md: "Leave unset [pid] to collect system-wide metrics only"). Metrics are sent to Sentinel when the repo secret SENTINEL_API_TOKEN is set (Settings → Secrets and variables → Actions → Secrets). Without the token, tracking runs locally only and Sentinel upload is skipped.

Job metadata (mirrors sc-inspector benchmark runs):

VariableCICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary default
TRACKER_PROJECT_NAMEsc-images
TRACKER_JOB_NAMEmatrix entry name, e.g. benchmark-vllm-gpu (arm64)
TRACKER_TASK_NAMEimage folder, e.g. benchmark-vllm-gpu
TRACKER_STAGE_NAMEworkflow name
TRACKER_EXTERNAL_RUN_IDgithub.run_id-github.run_attempt
TRACKER_CONTAINER_IMAGEghcr.io/sparecores/<folder>:main-<arch>
TRACKER_ORCHESTRATORgithub-actions
TRACKER_ENVci
TRACKER_QUIETtrue (no metric spam in build logs)

Scripts: install-resource-tracker.sh, run-with-resource-tracker.sh, build-push-image.sh.

CI Runners​

Push builds use GitHub-hosted runners (ubuntu-latest for amd64, ubuntu-24.04-arm for arm64). The exception is vllm-cpu-base-avx2, which always builds on [self-hosted, Linux, X64, workhorse] (build-level.yml).

Manual Run workflow checkboxes (default: off):

InputWhen enabled
self_hosted_x64amd64 → [self-hosted, Linux, X64]
self_hosted_arm64arm64 → [self-hosted, Linux, Arm64]

Images with a ZRAM metadata file run .github/scripts/setup-zram.sh before the build (PERCENT=125 by default → compressed swap ≈ 1.25× RAMRAM: Random-Access MemoryThe main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off.WikipediaGlossary). Currently enabled for vllm-cpu-base-avx2 (source-compiled CPU image) and benchmark-ffmpeg. On GitHub-hosted Azure kernels, zramzramA Linux kernel module that creates compressed block devices in memory. It is most often used as swap, so swapped-out pages are compressed in RAM instead of written to disk, trading CPU time for memory.Linux kernel docsGlossary is in linux-modules-extra-$(uname -r) (Launchpad #1762756); the script installs that package when needed. If the extra-modules deb is missing from apt (kernel/image drift), it keeps the default /swapfile. Tune the default percent via workflow env ZRAM_PERCENT, or override per image with a numeric ZRAM value.

vllm-cpu-base-avx2 compile parallelism is computed by .github/scripts/compute-build-parallelism.sh (RAM-linear on self-hosted, conservative on GitHub-hosted). GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary images use the multi-arch Hub vllm/vllm-openai base, so there is no in-repo CUDA source build.

vLLM Source Builds​

The following subsections apply only to source-built vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary images, currently vllm-cpu-base-avx2 on the self-hosted workhorse runner.

sccache​

Images with a SCCACHE metadata file compile C++/CUDA through sccache with an S3 backend (bucket in us-west-2, 90-day lifecycle).

Repo configuration (set by Pulumi after pulumi up):

NameKindPurpose
SCCACHE_BUCKETActions variableS3 bucket name
SCCACHE_REGIONActions variableBucket region (us-west-2)
SC_IMAGES_GHA_ROLE_ARNActions secretOIDC role for read/write on the bucket

Each build uses prefix {folder}/{arch}/ (e.g. vllm-cpu-base-avx2/amd64/) via SCCACHE_S3_KEY_PREFIX. Docker layer cache is exported to GHCR (buildcache-<arch>) and GitHub Actions cache (type=gha, scope {folder}-{arch} or {folder}-{arch}-v{VLLM_VERSION} for source-built vLLM CPU base). The sccache tool caches compiler objects inside the vLLM source compile steps.

Coverage when enabled:

StageMechanism
rust-buildRUSTC_WRAPPER=/usr/bin/sccache for cargo; CC/CXX absolute paths for build-script C compiles (openssl-sys, etc.)
csrc-build / vllm-build wheelCMake compiler launchers via setup.py (CMAKE_*_COMPILER_LAUNCHER=sccache); RUSTC_WRAPPER unset so C++ is not double-wrapped

Version 0.15.0 of sccache is installed to /usr/bin/sccache. The SCCACHE_IGNORE_SERVER_IO_ERROR=1 setting falls back to local compiles if the cache server is unreachable.

Currently enabled for vllm-cpu-base-avx2.

Sharing the Layer Cache Between GitHub- and Self-Hosted Runners​

A slow image can be pre-warmed on a self-hosted runner and then finished quickly on a GitHub-hosted one, but only if both produce identical BuildKit cache keys. Two things break that by default, and both are handled for source-built images:

  • Floating base tags and per-runner build args. ARG max_jobs and ENV CARGO_BUILD_JOBS are baked to the 64 GiB reference values and FROM ubuntu:22.04 is pinned by digest, so the Dockerfile is byte-identical everywhere (.github/scripts/tune-vllm-cpu-dockerfile.sh prints its sha256). Actual parallelism is injected at build time through the vllm_parallelism BuildKit secret, whose content is excluded from the cache key.
  • Build context ownership. BuildKit hashes local contexts with a V1 tarsum covering name, mode, uid, gid, size, and content. It excludes mtime but not uid/gid. GitHub-hosted runners check out as runner, self-hosted ones as ubuntu, so identical sources would hash differently and every COPY (plus the entire graph below it) would miss. .github/scripts/normalize-context-ownership.sh chowns the context to a canonical 0:0 before the build; build-level.yml reclaims workspace ownership on persistent runners before actions/checkout cleans it.

Changing any of the above invalidates the published buildcache-<arch>, so re-warm once on a self-hosted runner before relying on it from GitHub-hosted runners.

Layer Compression​

Published images use zstd layer compression (oci-mediatypes=true, force-compression=true) so pulls on sc-inspector VMsVM: virtual machineA computer emulated in software on a physical host, with its own OS and a share of the host's CPUs, memory, and storage. Most cloud servers are VMs.WikipediaGlossary decompress faster than gzipgzipA lossless file compressor of the GNU Project, first released in 1992, and its file format, specified in RFC 1952. It uses the DEFLATE algorithm, which combines LZ77 matching and Huffman coding, and is the most widely supported compression format on Unix-like systems and the web. Its levels range from 1 (fastest) to 9 (smallest output).GNU gzipGlossary (Docker's default). Requires Docker Engine ≥ 23.0 on pull clients. Implemented in build-push-image.sh; override with BUILD_COMPRESSION=gzip to revert to the legacy format.

CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary push compression runs inside BuildKit's Go zstd encoder (moby/buildkit#2345); there is no zstd -T equivalent exposed on docker buildx build. It uses limited block-level concurrency (tied to GOMAXPROCS in the buildkitd container), not libzstd's multi-threaded mode.