# Glossary

> The terms used in the docs of the benchmark-llm image.

Acronyms and terms used on this image's pages. See the [site-wide glossary](https://docs.sparecores.com/glossary.md) for all terms.

## A {#letter-a}

### AMD64 {#amd64}

The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it `amd64` or `x86_64`. See [Wikipedia](https://en.wikipedia.org/wiki/X86-64).

### Arm64 {#arm64}

The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it `arm64` or `aarch64`. See [Wikipedia](https://en.wikipedia.org/wiki/AArch64).

### AWS {#aws}

**Amazon Web Services**: The cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning. See [AWS](https://aws.amazon.com/).

## B {#letter-b}

### `-b` {#benchmark-llm-b}

The llama.cpp option that sets the logical batch size: the maximum number of prompt tokens passed to the model in one call. Each logical batch is split into physical batches of the size set by `-ub`, so a larger logical batch can process long prompts in fewer calls, at the cost of more memory. See [llama-bench README](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench).

## C {#letter-c}

### CDN {#cdn}

**Content Delivery Network**: A network of distributed servers that cache content, such as web pages, images or video, and serve it from locations close to its users. It lowers latency for the users and offloads traffic from the origin server. See [Wikipedia](https://en.wikipedia.org/wiki/Content_delivery_network).

### CI {#ci}

**Continuous Integration**: The practice of building and testing code automatically on every change, usually on a hosted service such as GitHub Actions, so problems show up before the change is merged. See [Wikipedia](https://en.wikipedia.org/wiki/Continuous_integration).

### container entrypoint {#container-entrypoint}

The executable that a container image runs when a container starts, set by the `ENTRYPOINT` instruction of its Dockerfile. The image's default command (`CMD`), or the arguments given after the image name in `docker run`, are passed to it as arguments, so an entrypoint can wrap any command, for example to set up the environment or record resource usage first. The `docker run --entrypoint` option replaces it. See [Dockerfile reference](https://docs.docker.com/reference/dockerfile/#entrypoint).

### CPU {#cpu}

**Central Processing Unit**: The general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads. See [Wikipedia](https://en.wikipedia.org/wiki/Central_processing_unit).

### CUDA {#cuda}

**Compute Unified Device Architecture**: The parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN. See [NVIDIA docs](https://docs.nvidia.com/cuda/).

## G {#letter-g}

### GGML {#ggml}

**Georgi Gerganov Machine Learning**: An open-source tensor library for machine learning, written in C and C++ without external dependencies. It runs on CPUs of many architectures, with SIMD-optimized code, and on GPUs and other accelerators through backends, and it supports quantized weights of 2 to 8 bits. llama.cpp and the GGUF file format are built on it. See [ggml on GitHub](https://github.com/ggml-org/ggml).

### GGUF {#gguf}

**GGML Universal File**: A binary file format for LLM weights and metadata, introduced in 2023 by the ggml project behind llama.cpp. One file holds the model's tensors, often quantized, together with its tokenizer and settings, so a model can be loaded quickly with memory mapping and without other files. Hugging Face hosts many models in this format. See [GGUF specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md), [Hugging Face docs](https://huggingface.co/docs/hub/gguf).

### GPU {#gpu}

**Graphics Processing Unit**: A processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory. See [Wikipedia](https://en.wikipedia.org/wiki/Graphics_processing_unit).

## H {#letter-h}

### harness {#harness}

The code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures. See [Wikipedia](https://en.wikipedia.org/wiki/Test_harness).

### Hugging Face {#hugging-face}

A company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models. See [Hugging Face Hub docs](https://huggingface.co/docs/hub).

## J {#letter-j}

### JSON {#json}

**JavaScript Object Notation**: A text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output. See [JSON](https://www.json.org/).

### JSON Lines {#json-lines}

A text format that stores one JSON value, usually an object, per line, so a file can be written and read one record at a time. It suits logs and streamed results, and line-based tools such as grep can process it. Files in this format usually end in `.jsonl`. See [JSON Lines](https://jsonlines.org/).

## L {#letter-l}

### `llama-bench` {#llama-bench}

The benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines. See [llama-bench README](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench).

### llama.cpp {#llama-cpp}

An open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, `llama-bench`. See [llama.cpp on GitHub](https://github.com/ggml-org/llama.cpp).

### LLM {#llm}

**Large Language Model**: A neural network, typically with billions of parameters, trained on large amounts of text to predict the next token. It can generate, summarize and translate text, answer questions and write code. See [Wikipedia](https://en.wikipedia.org/wiki/Large_language_model).

### LLM inference speed {#llm-inference-speed}

How fast hardware runs an LLM, usually measured in tokens per second. It is measured separately for prompt processing, in which the model reads the input prompt, and text generation, in which it produces output tokens one at a time, because the first is usually limited by compute and the second by memory bandwidth. Benchmarks of LLM servers also measure latency, such as the time to first token, and the throughput of many concurrent requests.

## N {#letter-n}

### `-ngl` {#benchmark-llm-ngl}

The llama.cpp option that sets how many layers of a model are offloaded to the GPUs. The remaining layers run on the CPU, so a model that does not fit in GPU memory can still use the GPU in part, and a value higher than the model's number of layers, such as 999, offloads all of them. See [llama-bench README](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench).

### `nice` {#nice}

A Unix command that runs a program with a changed scheduling priority, from -20 (highest) to 19 (lowest). Raising a process's priority above the default needs the `CAP_SYS_NICE` capability, which root has. See [man page](https://man7.org/linux/man-pages/man1/nice.1.html).

### NVIDIA Container Toolkit {#nvidia-container-toolkit}

A set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container. See [NVIDIA docs](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html).

### NVIDIA driver {#nvidia-driver}

The kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images. See [NVIDIA docs](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html).

## P {#letter-p}

### prompt processing {#prompt-processing}

The first phase of LLM inference, also called prefill, in which the model reads all tokens of the input prompt and stores their intermediate results in its key-value cache. The prompt's tokens are processed together in batches, so this phase is usually limited by compute and runs much faster per token than text generation. Its speed decides how long a long input takes before the first output token appears.

### `psutil` {#psutil}

A cross-platform Python library for retrieving information on running processes and system utilization, such as CPU, memory, disks, and network. It can count both logical CPUs and physical cores. See [psutil docs](https://psutil.readthedocs.io/).

## Q {#letter-q}

### `Q4_K_M` {#benchmark-llm-q4-k-m}

A llama.cpp quantization type of the k-quant family. It stores most weights with about 4 bits each, in blocks that share scale factors, and keeps some tensors at higher precision. The M (medium) variant is a common compromise between file size and output quality. See [Hugging Face GGUF docs](https://huggingface.co/docs/hub/gguf).

### quantization {#quantization}

Storing the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy. See [Hugging Face docs](https://huggingface.co/docs/optimum/concept_guides/quantization).

## R {#letter-r}

### RAM {#ram}

**Random-Access Memory**: The main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off. See [Wikipedia](https://en.wikipedia.org/wiki/Random-access_memory).

### `resource-tracker` {#resource-tracker}

An open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload. See [resource-tracker on GitHub](https://github.com/SpareCores/resource-tracker-rs).

## S {#letter-s}

### `SIGINT` {#sigint}

The interrupt signal of POSIX systems, which a terminal sends to the running process when the user presses Ctrl+C. A process can catch it to clean up before it exits; by convention, a process that exits because of a signal returns 128 plus the signal number, 130 for SIGINT. See [signal(7) man page](https://man7.org/linux/man-pages/man7/signal.7.html).

### `SIGTERM` {#sigterm}

The termination signal of POSIX systems, which asks a process to exit cleanly. It is the default signal of the `kill` command, and `docker stop` sends it to a container's main process first, then SIGKILL, which cannot be caught, if the process has not exited after a grace period. By convention, a process that exits because of it returns 143 (128 plus 15). See [signal(7) man page](https://man7.org/linux/man-pages/man7/signal.7.html).

### `-sm` {#benchmark-llm-sm}

The llama.cpp option that sets the split mode: how a model is distributed across several GPUs. The `layer` mode assigns whole layers to each GPU, `row` splits the weight tensors of each layer across the GPUs, and `none` uses a single GPU. See [llama-bench README](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench).

### stderr {#stderr}

The standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program. See [Wikipedia](https://en.wikipedia.org/wiki/Standard_streams).

### stdout {#stdout}

The standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as `> results.jsonl`, while logs on stderr still reach the terminal. See [Wikipedia](https://en.wikipedia.org/wiki/Standard_streams).

## T {#letter-t}

### text generation {#text-generation}

The second phase of LLM inference, also called decoding, in which the model produces output tokens one at a time, each depending on the tokens before it. Every step reads all model weights and the key-value cache, so this phase is usually limited by memory bandwidth rather than compute, and runs much slower per token than prompt processing.

### token {#token}

The unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.

## U {#letter-u}

### `-ub` {#benchmark-llm-ub}

The llama.cpp option that sets the physical batch size: the number of tokens computed at once within a logical batch (`-b`). It trades memory use against the speed of prompt processing. See [llama-bench README](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench).

### URL {#url}

**Uniform Resource Locator**: The address of a resource on the web or a network, such as https://sparecores.com/, made of a scheme, a host, and a path.

## V {#letter-v}

### vLLM {#vllm}

**virtual Large Language Model**: An open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API. See [vLLM docs](https://docs.vllm.ai/), [vLLM on GitHub](https://github.com/vllm-project/vllm), [Origin of the name](https://github.com/vllm-project/vllm/issues/835).
