Skip to main content

Usage

Shared Memory and Versions​

The vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary benchmarks fail with Docker's default 64 MiB /dev/shm, so pass --shm-size=4g. On CPUsCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary, when it runs in Docker, the harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary logs a warning if /dev/shm is smaller than 1 GiB; on GPUs, it does not warn before it starts. When a vLLM server fails to start on CPUs, the error in the log suggests --privileged and --shm-size=4g. Only --shm-size=4g is required; --privileged matches production.

Each image's Dockerfile sets BENCHMARK_VLLM_MODE, so the harness always runs in that image's mode: gpu in benchmark-vllm-gpu, even without GPUs, and cpu in the CPU images, even on a server with GPUs. Each image checks its hardware at startup; see its page.

The --version option prints the pinned vLLM and GuideLLMGuideLLMAn open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports.GuideLLM on GitHubGlossary versions and exits, such as:

docker run --rm ghcr.io/sparecores/benchmark-vllm-cpu:main --version

It prints vllm=0.27.1 guidellm=0.7.3, read from the version files built into the image. All three images take their versions from the same files in vllm-common/.

Hugging Face Access and Model Cache​

The container needs network access to Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary: the harness reads each model's metadata from it, and the model weights are downloaded from it. Pass an access tokenaccess tokenA secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted.Hugging Face Hub docsGlossary in HF_TOKEN, which the harness uses for both; production passes the host's access token too.

The default models gemma-2b and llama-8b are gated modelsgated modelA model on the Hugging Face Hub whose files can be downloaded only after a logged-in user accepts the conditions set by its authors, such as a license. Programs that download it then need an access token of an account that has accepted them.Hugging Face docsGlossary, so they need an access token from an account that has accepted their licenses. Without one, the harness cannot read a gated model's metadata: it logs metadata lookup failed for <model_id>, skips the model unless BENCHMARK_VLLM_ALLOW_ESTIMATED_METADATA is set, and continues with the next model.

The images set HF_HOME and HUGGINGFACE_HUB_CACHE to /models, a Docker volume, so a plain run downloads the models again each time. To reuse them across runs, mount a host directory on it with -v, such as:

docker run --rm --shm-size=4g -e HF_TOKEN -v "$PWD/models:/models" ghcr.io/sparecores/benchmark-vllm-cpu:main > results.jsonl 2> results.log

If the --models-dir directory (default: /models) has less than 1 GB of free space at startup, the harness exits with code 1. This is its only disk check: it never deletes the models it downloads, so every model of a run, up to the largest one that fits, stays in that directory. Make sure it has room for all of them. It also caches the memory footprint it computes for each model in .vllm-footprints/ under that directory.

Run Duration​

A run has a 2-hour budget, which it splits across the runnable models and workloads; see Time Budget. Sample production runs took the following times:

Server typeImageDuration
AWSAWS: Amazon Web ServicesThe cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning.AWSGlossary (Amazon Web Services) c8a.largebenchmark-vllm-cpu28 minutes (run)
AWS c8a.16xlargebenchmark-vllm-cpu1 hour 5 minutes (run)
AWS g5.16xlarge, one NVIDIA A10GNVIDIA A10GA data center GPU of NVIDIA's Ampere generation with 24 GB of GDDR6 memory and compute capability 8.6, a variant of the NVIDIA A10 made for AWS. It powers the EC2 G5 instances, which are used for machine learning inference and graphics workloads.AWSGlossary GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossarybenchmark-vllm-gpu1 hour 14 minutes (run)
UpCloud GPU-32xCPU-384GB-3xL40S, three NVIDIA L40SNVIDIA L40SA data center GPU of NVIDIA's Ada Lovelace generation with 48 GB of GDDR6 memory and compute capability 8.9. It is built for AI inference and training as well as graphics and video workloads.NVIDIAGlossary GPUsbenchmark-vllm-gpu1 hour 40 minutes (run)

The --benchmark-timeout-scale option takes a whole number of 1 or more, and mainly extends the time limit of each GuideLLM process. It does not let a run go past 2 hours; see Time Budget.

Plan and Probe Modes​

The following options run part of the harness only:

  • --plan-only prints one JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary object with the host's resources, such as its vCPUsvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary, memory, NUMANUMA: Non-Uniform Memory AccessA memory design where each CPU socket or chiplet has its own local memory, which its cores reach faster than the memory of the others. Software that keeps its threads and their memory on the same NUMA node avoids the slower remote accesses.Linux kernel docsGlossary (Non-Uniform Memory Access) nodes, and GPUs. For each model, the object shows whether it fits in memory at the shortest workload context and, if not, why; see CPU Memory Fit and GPU Memory Fit. It reads the model metadata from Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary but starts no vLLM server. On CPUs, it reads the host's memory, not the container's --memory limit. For example:

    docker run --rm -e HF_TOKEN ghcr.io/sparecores/benchmark-vllm-cpu:main --plan-only
  • --probe-only starts the smallest model, set by VLLM_PROBE_MODEL (default: smol-135m), and waits for the server's health check, for up to 15 minutes on CPUs and 8 minutes on GPUs. It exits with code 0 if the server started, and 1 otherwise. Production uses it to choose the vLLM image; see Production Setup.

Output Fields​

Each JSON line has the main fields listed in Results. The following fields are specific to CPU or GPU runs, or read differently on each:

FieldOn CPUsOn GPUs
modecpugpu
avx512Whether an AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary CPU has AVX-512AVX-512: Advanced Vector Extensions 512A set of x86 SIMD instructions that operate on 512-bit registers, so that one instruction processes, for example, 16 single-precision floating-point numbers at once. Intel introduced it in its server CPUs with the Skylake generation, and AMD supports it from the Zen 4 generation. It is made of several subsets, such as the foundation, avx512f, and avx512_bf16 for bfloat16 arithmetic, which Linux lists as flags in /proc/cpuinfo.WikipediaGlossary (Advanced Vector Extensions 512); null on Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossaryDescribes CPU runs; ignore it
avx2_only_imagetrue only when BENCHMARK_VLLM_ALLOW_AVX2_ONLY is set, as in benchmark-vllm-cpu-avx2Describes CPU runs; ignore it
tensor_parallel, data_parallelThe model's split across the NUMA nodes; see CPU ParallelismThe model's split across the GPUs; see GPU Parallelism
pipeline_parallelAlways 1The GPU count for llama-70b on more than one GPU, while tensor_parallel still holds the size that the head-count rule gives
omp_threads_bindThe CPUs of each vLLM worker, separated by vertical bars, such as 0-15|16-31|32-46 for three workersnull
gpu_count0The number of GPUs
gpu_model, total_vram_gbDescribe GPU runs; ignore themThe name of the first GPU, and the memory of all GPUs together in GiB
tuningThe GuideLLM load and the vLLM server settings, passed to vllm serve; see Load SettingsThe GuideLLM load; the vLLM server settings are computed but not passed to vllm serve

The following fields read the same in both modes:

  • tuning holds the GuideLLM load, such as max_concurrency and sweep_size, and the vLLM server settings max_num_seqs, max_num_batched_tokens, dtype, and kv_memory_util. Its gpu_memory_util value applies to GPU runs only.
  • tuning_version is the version of the rules that compute these settings, 9 in the current harness, and 0 with BENCHMARK_VLLM_AUTOCONFIG=0.
  • model_weight_gib is the size of the model weights in GiB, model_torch_dtype the data type in the model's configuration, and model_metadata_source where the harness read the metadata from. The kv_bytes_per_token_per_tp_rank field is the key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary size of one tokentokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary on each vLLM worker of a tensor-paralleltensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary group in bytes.
  • runtime_memory holds the key-value cache size per vLLM worker in GiB and in tokens, and the --kv-cache-memory-bytes--kv-cache-memory-bytesThe vLLM option that sets the size of the key-value cache of each worker in bytes. Without it, vLLM derives the cache size from --gpu-memory-utilization; this option gives finer control over how much memory the cache takes.vLLM engine argumentsGlossary value that vLLM suggests, as read from the server's log. It is empty when the log has none of them.

The scores cover only the requests that succeeded. For a metric that GuideLLM gives no separate statistics of successful requests for, they cover all requests, failed ones included. The output does not record how many requests failed.

Exit Codes​

The harness exits with one of the following codes:

  • 0: it wrote at least one JSON line, even if it skipped models or workloads, or stopped early.
  • 1: one of the following cases applies:
    • The --models-dir directory has less than 1 GB of free space.
    • The guidellm command is missing.
    • No model could run, for example because none fits in memory.
    • No JSON line was written.
    • On GPUs, every visible GPU has a compute capabilitycompute capabilityA version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later.NVIDIAGlossary below 8.0; see benchmark-vllm-gpu.
  • 2: on CPUs, the CPU is an AMD64 CPU without AVX-512, unless BENCHMARK_VLLM_ALLOW_AVX2_ONLY is set; see benchmark-vllm-cpu.

Check which lines are present in stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary, and look for the following messages in stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary:

  • metadata lookup failed for <model_id>: the model's Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary metadata could not be read; see Hugging Face Access and Model Cache.
  • Skipping <model> workload=<name> — insufficient KV cache headroom: on CPUs, the workload's key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary did not fit in memory; see CPU Memory Fit.
  • Overall timeout reached: the run reached its 2-hour stop, and the remaining models did not run; see Time Budget.

Production Setup​

The Inspector of Spare Cores Navigator runs one task, vllm, for the three vLLM images, on every server type except those on the inspector_rules.py blocklist. The blocklist also excludes server types with less than 2 GB of memory from this task. The task uses the following settings:

SettingProductionPlain docker run
Imagebenchmark-vllm-gpu first, then benchmark-vllm-cpu, then benchmark-vllm-cpu-avx2. The GPU image is skipped on servers without GPUs or with every GPU below compute capabilitycompute capabilityA version number, such as 8.0 or 9.0, that NVIDIA assigns to each GPU generation to identify the hardware features and instructions it supports. The major number marks the architecture, for example 7 for Volta and Turing, 8 for Ampere and Ada Lovelace, and 9 for Hopper. CUDA software can require a minimum compute capability, so that it runs only on GPUs of that generation or later.NVIDIAGlossary 8.0, and benchmark-vllm-cpu-avx2 on Arm64. Each image first runs --probe-only, for at most 18 minutes, and the first that exits with code 0 runs the full benchmark, with no fallback after that.Only the image passed to docker run, with no --probe-only step first and no fallback to another image
GPUsAll GPUsAs passed with --gpus
Shared memory4 GiB64 MiB, unless --shm-size is passed
PrivilegesPrivilegedNot privileged, unless --privileged is passed
NetworkingHost networkingDocker's default bridge network, unless --network host is passed
Memory limit85% of the host's available memory, with swap disabled and a swappinessswappinessA Linux kernel setting, vm.swappiness, that sets how readily the kernel swaps out process memory compared with dropping cached file data when it needs free memory. Higher values swap more eagerly; 0 makes the kernel avoid swapping as long as it can. Container runtimes such as Docker can also set it per container.Linux kernel docsGlossary of 0None, unless --memory is passed, and swap as the host allows
EnvironmentThe host's HF_TOKEN and the resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary settingsAs passed with -e, such as -e HF_TOKEN, without the resource-tracker settings
Overall timeout3 hoursNone, only the harness's own 2-hour budget
Other tasksNot run in parallel with other benchmarksWhatever else runs on the host at the same time

To reproduce the memory limit, pass --memory=<limit> --memory-swap=<limit> --memory-swappiness=0, with <limit> set to 85% of the host's available memory; the same value for --memory-swap disables swap. Production reads the memory that Linux reports as available, the available column of free -g in GiB. Write the limit with a unit: for example, with 40 GiB available, pass --memory=34g --memory-swap=34g.

Production stores the results of all three images under the task name vllm, in the data/<vendor>/<server>/<variant>/vllm/ folder of sc-inspector-data, where <variant> is a numbered folder, 0 for most server types. The task tries the fastest image first, so the result of a server type comes from the fastest image that runs on it.

We investigate failed production runs, manually or semi-automatically, and then run them again. Until then, a server type's latest result in sc-inspector-data can be a failed or partial run.

Options​

Pass the following options after the image name. See Environment Variables for the settings passed with -e.

OptionMeaningDefault
--modelsHugging Face model IDsHugging Face model IDThe name of a model repository on the Hugging Face Hub, in the form <owner>/<model>, such as Qwen/Qwen2.5-0.5B-Instruct. Programs and libraries use it to download the model's files, and it appears as the title of the model's page on the Hub.Hugging Face Hub docsGlossary, or short names of the default models, to run instead of the default models, separated by spaces, such as qwen-0.5b <owner>/<model>The six default models on GPUs, and the five that run on CPUs
--models-dirDirectory of the 1 GB free-space check and of the model metadata that the harness downloads. The model weights still go to HF_HOME and HUGGINGFACE_HUB_CACHE (/models), so mount a disk at /models to move them/models
--benchmark-timeout-scaleWhole number of 1 or more that multiplies the time limits of GuideLLM, but not the 2-hour stop; see Time Budget1
--probe-onlyStart the probe model, wait for its health check, and exitOff
--plan-onlyPrint the host's resources and whether each model fits in memory, as JSON, without starting vLLMOff
--versionPrint the pinned vLLM and GuideLLM versions and exitOff

Environment Variables​

A default marked "legacy" applies with BENCHMARK_VLLM_AUTOCONFIG=0; see Legacy Settings. If a row lists two variables and both are set, the first wins, except in the HF_HOME and HUGGINGFACE_HUB_CACHE row: the Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary libraries read those, not the harness. The harness does not check the GPU tensor-paralleltensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary override against the GPU count or the head count of the model.

VariableMeaningDefault
HF_TOKENHugging Face access tokenaccess tokenA secret string that a program sends to an online service to act for an account, in place of the account's password. The token's permissions set what the program may do; a Hugging Face read token, for example, lets a program download the gated models whose licenses the account has accepted.Hugging Face Hub docsGlossary for the model metadata and downloadsUnset
BENCHMARK_VLLM_MODEgpu or cpu modeSet by each image
BENCHMARK_VLLM_WORKLOADSComma-separated workload names; a value that matches no workload of the mode, such as long on CPUs, runs all of themchat,rag,long on GPUs, chat,rag on CPUs
BENCHMARK_VLLM_AUTOCONFIG0, false, or no uses the legacy static settings1
BENCHMARK_VLLM_PER_WORKLOAD_SERVERRestart vLLM for each workload with its context length; 0, false, or no turns this off, so one server with the context length of the longest workload runs all workloadsSame as BENCHMARK_VLLM_AUTOCONFIG
BENCHMARK_VLLM_ALLOW_ESTIMATED_METADATAAny non-empty value, even 0, runs a model with the size estimated from the parameter count in the default model table when its Hugging Face metadata cannot be readUnset, so the model is skipped
BENCHMARK_VLLM_VCPUSHost vCPU count used for the load settingsThe CPUs the process may run on
BENCHMARK_VLLM_MAX_NUM_SEQSPassed to vLLM as --max-num-seqs--max-num-seqsThe vLLM option that sets the maximum number of sequences, or requests, that a vLLM engine processes together in one scheduling step. Further requests wait in a queue until a running sequence finishes. With data parallelism, each replica applies the limit on its own.vLLM engine argumentsGlossaryComputed on CPUs; unset on GPUs, so vLLM's default
BENCHMARK_VLLM_MAX_NUM_BATCHED_TOKENSPassed to vLLM as --max-num-batched-tokens--max-num-batched-tokensThe vLLM option that sets the maximum number of tokens that a vLLM engine processes in one scheduling step, summed over all sequences in the batch. Prompt tokens and generated tokens both count toward it, so a lower value limits the work and memory of each step, and a higher value lets more prompt tokens be processed at once.vLLM engine argumentsGlossaryComputed on CPUs; unset on GPUs, so vLLM's default
BENCHMARK_VLLM_RAMPUPRamp-up of the throughput stage in secondsComputed
BENCHMARK_VLLM_ALLOW_AVX2_ONLYOn CPUs: 1, true, or yes skips the AVX-512 check on AMD64Unset; true in benchmark-vllm-cpu-avx2
BENCHMARK_VLLM_CPU_TP, VLLM_CPU_TENSOR_PARALLEL_SIZEOn CPUs: tensor-parallel sizeThe NUMA node count if it divides the model's attention headsattention headOne of several parallel attention computations in a layer of a transformer model. Each head projects the tokens into its own smaller query, key, and value vectors and learns which earlier tokens to focus on, and the layer combines the outputs of all heads. The number of heads per layer is fixed by the model's architecture, and methods that split a layer across devices, such as tensor parallelism, usually assign whole heads to each device.Attention Is All You NeedGlossary, otherwise 1
BENCHMARK_VLLM_CPU_DP, VLLM_CPU_DATA_PARALLEL_SIZEOn CPUs: data-paralleldata parallelismA way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices.Hugging Face docsGlossary sizeComputed; see CPU Parallelism
VLLM_CPU_OMP_THREADS_BINDVLLM_CPU_OMP_THREADS_BINDThe environment variable of vLLM's CPU backend that sets which CPU cores the OpenMP threads of each worker run on. It takes CPU lists such as 0-31, one per worker, separated by vertical bars, so 0-31|32-63 binds two workers to 32 cores each. The value auto binds each worker to the cores of one NUMA node, and nobind turns the binding off.vLLM CPU installationGlossaryOn CPUs: CPUs of each vLLM worker; any value other than auto is passed to vLLM as isauto, set by the CPU images, so the harness computes it
BENCHMARK_VLLM_NUMA_NODESOn CPUs: NUMA node countRead from the host
BENCHMARK_VLLM_CPU_TOPOLOGYOn CPUs: logical CPUs of each NUMA node, as CPU lists separated by vertical barsRead from the host
BENCHMARK_VLLM_CPU_PHYSICAL_TOPOLOGYOn CPUs: one logical CPU per physical core of each NUMA node, in the same formatRead from the host
VLLM_CPU_GPU_MEMORY_UTILIZATIONOn CPUs: value of --gpu-memory-utilization--gpu-memory-utilizationThe vLLM option that sets the fraction of device memory, from 0 to 1, that a vLLM worker may use for the model weights, intermediate results, and key-value cache. On GPUs, it applies to the memory of each GPU; on the CPU backend of vLLM, each worker applies it to the total memory of its NUMA node, despite the option's name. The memory left after the weights and intermediate results goes to the key-value cache, so a higher value lets the server hold more concurrent sequences.vLLM engine argumentsGlossary for each vLLM workerComputed
VLLM_CPU_KVCACHE_SPACEOn CPUs: passed to vLLM, the key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary size of its CPU backend in GiBUnset
VLLM_CPU_DTYPEOn CPUs: value of --dtype--dtypeThe vLLM option that sets the data type of the model weights and activations, such as bfloat16 or float32. With its default, auto, vLLM derives the data type from the model's configuration. A smaller data type needs less memory and bandwidth per weight, but only runs fast on hardware that supports it.vLLM engine argumentsGlossarybfloat16 or float32, depending on the CPU
BENCHMARK_VLLM_GPU_TP, VLLM_GPU_TENSOR_PARALLEL_SIZEOn GPUs: tensor-parallel sizeFrom the GPU count and the model's attention heads
BENCHMARK_VLLM_GPU_DP, VLLM_GPU_DATA_PARALLEL_SIZEOn GPUs: data-parallel size1
VLLM_GPU_MEMORY_UTILIZATIONOn GPUs: value of --gpu-memory-utilization, also used by the memory fit check0.92
VLLM_PROBE_MODELModel of --probe-onlysmol-135m
VLLM_PROBE_HEALTH_TIMEOUT_CPU_SECOn CPUs: health check wait of --probe-only in seconds900
VLLM_PROBE_HEALTH_TIMEOUT_GPU_SECOn GPUs: health check wait of --probe-only in seconds480
GUIDELLM_CPU_SWEEP_SIZE, GUIDELLM_GPU_SWEEP_SIZE, GUIDELLM_SWEEP_SIZENumber of sweep stages, at least 2; the CPU or GPU variable applies in its modeComputed; legacy: 3
GUIDELLM__MAX_CONCURRENCYMaximum concurrent requestsComputed
GUIDELLM__MAX_WORKER_PROCESSESGuideLLM worker processesComputed
GUIDELLM_MAX_REQUESTS, GUIDELLM_MAX_REQUESTS_CPUAdds a request cap (max_requests constraint) to each GuideLLM stage; with BENCHMARK_VLLM_AUTOCONFIG=0, only GUIDELLM_MAX_REQUESTS_CPU applies on CPUsUnset; legacy: 25 on CPUs, 120 on GPUs
GUIDELLM_PROFILES, GUIDELLM_CPU_PROFILESAny of legacy, sync-throughput, or sync (all three do the same) runs one synchronous and one throughput stage instead of a sweep, with autoconfig on or off; the CPU variable applies on CPUs onlyUnset
GUIDELLM_THROUGHPUT_RATEConcurrency of the throughput stage; applies only with GUIDELLM_PROFILES or GUIDELLM_CPU_PROFILES8 on CPUs, 16 on GPUs
HF_HOME, HUGGINGFACE_HUB_CACHEHugging Face cache directory/models, set by the images
TRACKER_QUIETQuiet output of resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossarytrue, set by the images