Skip to main content

Contributor Guide

Image Layout​

The Dockerfile builds the image from the official llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary container images, at the build set by LLAMA_CPP_BUILD:

  • AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary: the final stage is ghcr.io/ggml-org/llama.cpp:full-cuda-<build>, with its llama.cpp build moved to /llama_cpp_gpu. The CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary build from ghcr.io/ggml-org/llama.cpp:full-<build> is copied to /llama_cpp_cpu.
  • Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossary: the final stage is ghcr.io/ggml-org/llama.cpp:full-<build>, with its llama.cpp build moved to /llama_cpp_cpu. There is no CUDA build.

On AMD64, extract-shared-cpu-libs.sh copies every shared library that ldd reports for the CPU build's llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary into /llama_cpp_cpu/libs.

The image installs psutilpsutilA cross-platform Python library for retrieving information on running processes and system utilization, such as CPU, memory, disks, and network. It can count both logical CPUs and physical cores.psutil docsGlossary with pip, copies the resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary binary, and declares /models as a volume. The container entrypointcontainer entrypointThe executable that a container image runs when a container starts, set by the ENTRYPOINT instruction of its Dockerfile. The image's default command (CMD), or the arguments given after the image name in docker run, are passed to it as arguments, so an entrypoint can wrap any command, for example to set up the environment or record resource usage first. The docker run --entrypoint option replaces it.Dockerfile referenceGlossary is resource-tracker -- /benchmark.py, with TRACKER_QUIET=true.

Build Arguments​

ArgumentMeaningDefault
LLAMA_CPP_BUILDTag of the llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary container images. Both the AMD64 and the Arm64 images must exist for the tag.b4753
RESOURCE_TRACKER_IMAGEImage to copy the resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary binary from. CICI: Continuous IntegrationThe practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged.WikipediaGlossary passes the image for the target architecture, ghcr.io/sparecores/resource-tracker:main-<arch>.ghcr.io/sparecores/resource-tracker:main

CI builds the image for amd64 and arm64, after the resource-tracker image it depends on.

To build the image locally, run the following command from the root of the sc-images repository. The image folder is the build context:

docker build --build-arg LLAMA_CPP_BUILD=b4753 -t benchmark-llm images/benchmark-llm

To check the rebuilt image, print the llama.cpp commit hash:

docker run --rm benchmark-llm --version

Changing LLAMA_CPP_BUILD or the default models makes new results incomparable with the published ones, which would need a rerun on every server type; see Fixed Models and Build.