Skip to main content

Workloads

Prompt and Output Lengths​

Each workload sends synthetic prompts of a fixed length and asks for a fixed number of output tokenstokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary. The harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary starts vllm serve with a context length (--max-model-len--max-model-lenThe vLLM option that sets the context length of a model, the maximum number of tokens, prompt and output together, in one sequence. The key-value cache must have room for at least one sequence of this length, so a shorter context length needs less memory. Without the option, vLLM uses the length from the model's configuration.vLLM engine argumentsGlossary), the most tokens that one request can hold, prompt and output together. Each workload gets a context length that fits it, because a shorter one needs a smaller key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary:

WorkloadPrompt tokensOutput tokensContext lengthRuns on
chat2561282,048CPUsCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary and GPUs
rag1,0242564,096CPUs and GPUs
long4,0965128,192GPUs only

Set BENCHMARK_VLLM_WORKLOADS to run fewer workloads, such as BENCHMARK_VLLM_WORKLOADS=chat.

Models​

The six default models run from smallest to largest:

Short nameHugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary modelParametersAttention heads
smol-135mHuggingFaceTB/SmolLM2-135M-Instruct135 million9
qwen-0.5bQwen/Qwen2.5-0.5B-Instruct0.5 billion14
gemma-2bgoogle/gemma-2-2b-it2 billion8
llama-8bmeta-llama/Llama-3.1-8B-Instruct8 billion32
phi-4microsoft/phi-414 billion40
llama-70bunsloth/Llama-3.3-70B-Instruct-bnb-4bit70 billionFrom its config.json

The gemma-2b and llama-8b models are gated modelsgated modelA model on the Hugging Face Hub whose files can be downloaded only after a logged-in user accepts the conditions set by its authors, such as a license. Programs that download it then need an access token of an account that has accepted them.Hugging Face docsGlossary; see Hugging Face Access and Model Cache. The head counts above are fallbacks: the harness reads each model's head count from its config.json on Hugging Face first.

The llama-70b model is 4-bit quantizedquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary with bitsandbytesbitsandbytesAn open-source Python library that quantizes the weights of PyTorch models to 8 or 4 bits, so that large models need less GPU memory. Hugging Face Transformers and vLLM can load models quantized with it, and its 4-bit format is also used to fine-tune large models on a single GPU.bitsandbytes on GitHubGlossary, and vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary serves it with --quantization bitsandbytes --load-format bitsandbytes. The harness treats these serve options as GPU-only, so it never runs that model on CPUs, where five default models run.

The --models option replaces this list with Hugging Face model IDsHugging Face model IDThe name of a model repository on the Hugging Face Hub, in the form <owner>/<model>, such as Qwen/Qwen2.5-0.5B-Instruct. Programs and libraries use it to download the model's files, and it appears as the title of the model's page on the Hub.Hugging Face Hub docsGlossary or the short names above.

Run Sequence​

For each model, the harness first checks that the model fits in memory; see CPU Memory Fit and GPU Memory Fit. Then it runs the following steps for each workload:

  1. Computes the load settings for the workload's context length.
  2. Skips the workload if the model's memory plan for that context length does not fit.
  3. Starts vllm serve and waits for its health check, up to 10 minutes on CPUs and 5 minutes on GPUs.
  4. Runs one GuideLLM sweep, and converts its results into JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary lines.
  5. Stops the server: it sends SIGTERMSIGTERMThe termination signal of POSIX systems, which asks a process to exit cleanly. It is the default signal of the kill command, and docker stop sends it to a container's main process first, then SIGKILL, which cannot be caught, if the process has not exited after a grace period. By convention, a process that exits because of it returns 143 (128 plus 15).signal(7) man pageGlossary to the server's process group, and SIGKILL after 30 seconds.

The following rules skip models or stop the run early:

  • If a model does not fit in memory, it is skipped, and the run continues with the next model.
  • If a model's server does not pass its health check, the run stops, and no larger model is tried.
  • If a model's highest mean output throughput is above 0 but below 1 token per second, the run stops after that model.
  • If a GuideLLMGuideLLMAn open-source tool of the vLLM project for evaluating LLM deployments. It sends synthetic or real requests to an inference server over an OpenAI-compatible HTTP API, at a fixed concurrency, a fixed request rate, or a sweep of rates, and measures latency and throughput, such as the time to first token and output tokens per second. It writes its results as JSON, CSV, or HTML reports.GuideLLM on GitHubGlossary run fails or times out, the lines of its partial results are still written, if it saved any.

In all of these cases, the run exits with code 0 as long as it wrote at least one JSON line; see Exit Codes.

GuideLLM Sweep​

GuideLLM runs in the same container as the vLLM server and sends its requests to http://127.0.0.1:8000 through vLLM's OpenAI-compatible /v1/completions endpoint, so the results include no network latency between hosts. The prompts are synthetic text of the workload's token counts. GuideLLM generates them from a fixed random seed (42), the starting value of its random number generator, so every run sends the same prompts.

The sweep size, N, is the number of load stages in a GuideLLM sweepGuideLLM sweepA GuideLLM benchmark profile that measures a server across a range of loads. It first runs a synchronous stage, which sends one request at a time, and a throughput stage, which sends as many concurrent requests as allowed. It then runs the remaining stages, if any, at constant request rates between the rates of those two stages. The sweep size is the total number of stages.GuideLLM on GitHubGlossary. The output records each stage as a strategy:

  1. synchronous: one request at a time, for the baseline latency.
  2. throughput: as many concurrent requests as the maximum concurrency allows, for the peak throughput.
  3. constant: N - 2 stages at constant request rates between the synchronous and throughput stages.

Each stage stops at its time limit, and the throughput stage ramps up its concurrency first. No stage has a request cap unless GUIDELLM_MAX_REQUESTS or, on CPUs, GUIDELLM_MAX_REQUESTS_CPU sets one. The GuideLLM command and its options are in the harness.

For example, on an AWSAWS: Amazon Web ServicesThe cloud computing platform of Amazon, launched in 2006 and the largest public cloud by market share. It offers hundreds of services, from virtual machines (EC2) and object storage (S3) to managed databases and machine learning.AWSGlossary (Amazon Web Services) c8a.12xlarge CPU server, each sweep had a size of 3, with 80 seconds per stage (output).

On an AWS g5.16xlarge with one NVIDIA A10GNVIDIA A10GA data center GPU of NVIDIA's Ampere generation with 24 GB of GDDR6 memory and compute capability 8.6, a variant of the NVIDIA A10 made for AWS. It powers the EC2 G5 instances, which are used for machine learning inference and graphics workloads.AWSGlossary GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary, the chat sweep had a size of 6: one synchronous, one throughput, and four constant-rate stages (output).

Load Settings​

By default (BENCHMARK_VLLM_AUTOCONFIG=1), the harness uses the host's vCPUvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary count (v) to set the GuideLLM load for both CPU and GPU runs. It does not derive the load from GPU count, model, or memory. Set BENCHMARK_VLLM_VCPUS to use a different vCPU count. Each output line's tuning object records the settings below. See Environment Variables to override them.

On CPUs​

On CPUs, the harness uses the following formulas for the load, where workers, the number of vLLM workers, is the tensor-paralleltensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary size times the data-paralleldata parallelismA way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices.Hugging Face docsGlossary size:

  • Maximum concurrency: max(min_concurrency, int(6 * v ** 0.65)), where min_concurrency is max(1, min(32, 4 * v), 4 * workers)
  • GuideLLM worker processes: max(1, min(32, v, max_concurrency // 8))
  • Sweep size: max(2, min(4, 2 + floor(log2(v) / 3))), that is, 2 below 8 vCPUs, 3 from 8 to 63 vCPUs, and 4 from 64 vCPUs, then reduced, down to 2, while the stages would be shorter than their minimum
  • Seconds per stage: the time limit of the sweep divided by the sweep size, at most 240 seconds, and at least 25 seconds for chat and 44 seconds for rag, a minimum that grows with the context length
  • Throughput-stage ramp-up: 15% of the seconds per stage, between 2s and 8s, and short enough to leave at least 25s of the stage at peak concurrency
  • Warmup: 10, passed to GuideLLM as warmup

On CPUs, the harness also computes the following vLLM server settings and passes them to vllm serve:

  • --max-num-seqs--max-num-seqsThe vLLM option that sets the maximum number of sequences, or requests, that a vLLM engine processes together in one scheduling step. Further requests wait in a queue until a running sequence finishes. With data parallelism, each replica applies the limit on its own.vLLM engine argumentsGlossary: the most requests that vLLM processes at once, with the rest waiting in a queue; it grows sub-linearly with v, and is capped by the CPU cores of each vLLM worker and by the key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary that fits in memory
  • --max-num-batched-tokens--max-num-batched-tokensThe vLLM option that sets the maximum number of tokens that a vLLM engine processes in one scheduling step, summed over all sequences in the batch. Prompt tokens and generated tokens both count toward it, so a lower value limits the work and memory of each step, and a higher value lets more prompt tokens be processed at once.vLLM engine argumentsGlossary: the most tokens, prompt and generated together, that vLLM processes in one step: min(max_model_len, max(2048, int(64 * v ** 0.5))), where max_model_len is the workload's context length
  • --dtype--dtypeThe vLLM option that sets the data type of the model weights and activations, such as bfloat16 or float32. With its default, auto, vLLM derives the data type from the model's configuration. A smaller data type needs less memory and bandwidth per weight, but only runs fast on hardware that supports it.vLLM engine argumentsGlossary: the data type; see Data Type
  • --gpu-memory-utilization--gpu-memory-utilizationThe vLLM option that sets the fraction of device memory, from 0 to 1, that a vLLM worker may use for the model weights, intermediate results, and key-value cache. On GPUs, it applies to the memory of each GPU; on the CPU backend of vLLM, each worker applies it to the total memory of its NUMA node, despite the option's name. The memory left after the weights and intermediate results goes to the key-value cache, so a higher value lets the server hold more concurrent sequences.vLLM engine argumentsGlossary: the memory fraction of each vLLM worker; see CPU Memory Fit
  • --tensor-parallel-size and --data-parallel-size: see CPU Parallelism

On GPUs​

On GPUs, the harness uses the following formulas for the load:

  • Maximum concurrency: max(64, int(8 * v ** 0.60))
  • GuideLLM worker processes: min(max(8, int(3 * v ** 0.50)), max(4, v))
  • Sweep size: max(2, min(3 + floor(log2(v) / 2), 6 + floor(log2(v) / 3))), then reduced until the sweep fits the time budget of the run
  • Seconds per stage: the run budget divided by the sweep size, at least max(25, 22 * max_model_len / 2048), and at most 240
  • Throughput-stage ramp-up in seconds: min(10, max(2, 0.15 * seconds per stage))
  • Warmup: 0.05, so GuideLLM leaves the first 5% of each stage out of the results, to exclude startup effects (GuideLLM)

Production runs used the following settings. The workload columns give the sweep size times the seconds per stage:

Server typevCPUsMaximum concurrencyWorkerschatraglong
AWS g5.16xlarge6497246 x 40s5 x 48s2 x 120s
AWS g5.8xlarge3264165 x 48s5 x 48s2 x 120s

The long workload also ran two stages of 120 seconds on the UpCloud GPU-32xCPU-384GB-3xL40S (output). Its minimum of 88 seconds per stage leaves room for only two stages in a run budget of 240 seconds.

On GPUs, the harness also computes the vLLM server settings max_num_seqs, max_num_batched_tokens, dtype, and kv_memory_util, and records them in the tuning object, but does not pass them to vllm serve, which uses its own defaults. Set BENCHMARK_VLLM_MAX_NUM_SEQS or BENCHMARK_VLLM_MAX_NUM_BATCHED_TOKENS to pass --max-num-seqs or --max-num-batched-tokens.

Time Budget​

A run has an overall budget of 2 hours. The harness splits it as follows:

  1. It reserves 600 seconds, plus time for each server start, one per runnable model and workload: 120 seconds on CPUs and 60 seconds on GPUs.
  2. It divides the rest by the number of sweeps, one per model and workload.
  3. It limits each sweep to 45 to 240 seconds.

For example, on the g5.16xlarge, the log shows overall=7200s reserve=1320s runs=12 per_run=240s (log).

Before it starts each model, workload, and GuideLLM process, the harness checks the elapsed time. After 2 hours, it starts nothing new: it logs "Overall timeout reached" before the next model, and the models it has not reached do not run. This can happen, for example, when the 45-second minimum per sweep pushes the plan past 2 hours.

The --benchmark-timeout-scale option takes a whole number, and values below 1 act as 1. It multiplies the following limits:

  • The overall budget, before the split above. Each sweep stays limited to 240 seconds, so its stages get longer only when its budget was below 240 seconds without the scale.
  • The time limit of each GuideLLM process.
  • With BENCHMARK_VLLM_AUTOCONFIG=0, the time limit of each stage.

The option does not scale the 2-hour stop, so a value above 1 does not let a run go on longer.

CPU Parallelism​

On CPUs, the harness does not run one vLLM worker across the whole host. It splits each model across the NUMANUMA: Non-Uniform Memory AccessA memory design where each CPU socket or chiplet has its own local memory, which its cores reach faster than the memory of the others. Software that keeps its threads and their memory on the same NUMA node avoids the slower remote accesses.Linux kernel docsGlossary (Non-Uniform Memory Access) nodes and CPU cores as follows:

  • Tensor parallelismtensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary: The tensor-parallel size is the NUMA node count when that count divides the model's attention headattention headOne of several parallel attention computations in a layer of a transformer model. Each head projects the tokens into its own smaller query, key, and value vectors and learns which earlier tokens to focus on, and the layer combines the outputs of all heads. The number of heads per layer is fixed by the model's architecture, and methods that split a layer across devices, such as tensor parallelism, usually assign whole heads to each device.Attention Is All You NeedGlossary count, and 1 otherwise.
  • Data parallelismdata parallelismA way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices.Hugging Face docsGlossary: The data-parallel size is the number of model replicas, each with a full copy of the weights. The harness sets it as follows:
    • At least one replica per NUMA node when the tensor-parallel size is below the NUMA node count.
    • On every host, more replicas if the physical core count, divided by the tensor-parallel size and by the cores per worker for the model size (see the table below), gives more.
    • At most 32 replicas, and only as many as fit in the memory of each NUMA node.
  • Thread bindingthread bindingRestricting each thread of a program to chosen CPU cores by setting its CPU affinity, also called CPU pinning. It keeps a thread's data in the caches and local memory of its cores, and stops the operating system from moving it between cores during a run, which makes performance faster and more repeatable, especially on hosts with several NUMA nodes.Linux man pageGlossary: The harness sets VLLM_CPU_OMP_THREADS_BINDVLLM_CPU_OMP_THREADS_BINDThe environment variable of vLLM's CPU backend that sets which CPU cores the OpenMP threads of each worker run on. It takes CPU lists such as 0-31, one per worker, separated by vertical bars, so 0-31|32-63 binds two workers to 32 cores each. The value auto binds each worker to the cores of one NUMA node, and nobind turns the binding off.vLLM CPU installationGlossary to an explicit CPU list for each worker, with one logical CPU per physical core, all from one NUMA node. It finds the SMTSMT: Simultaneous MultithreadingA CPU technique that runs two (or more) hardware threads on one physical core, sharing its execution units, e.g. Intel Hyper-Threading. It raises throughput when a thread stalls, but the threads compete for the core. Cloud vendors usually sell each thread as a vCPU.WikipediaGlossary (Simultaneous Multithreading) siblings of each core in sysfs (thread_siblings_list) and leaves them out. On hosts with at least 2 vCPUs, it keeps one CPU free for the server's frontend process. A VLLM_CPU_OMP_THREADS_BIND value other than auto replaces these lists.

The cores per worker depend on the model size:

Model sizePhysical cores per worker
Up to 0.5 billion parameters16
Up to 2 billion parameters32
Up to 8 billion parameters64
Larger128

For example, on the c8a.12xlarge, smol-135m ran as three replicas: 48 physical cores divided by 16 cores per worker. They were bound to 0-15|16-31|32-46, that is, 16, 16, and 15 cores, with one CPU left for the frontend (output).

GPU Parallelism​

On GPUs, the harness does not always use every GPU of the server. It splits each model across the GPUs as follows:

  • Tensor parallelismtensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary: The tensor-parallel size is the largest value, up to the GPU count, that divides the model's attention headattention headOne of several parallel attention computations in a layer of a transformer model. Each head projects the tokens into its own smaller query, key, and value vectors and learns which earlier tokens to focus on, and the layer combines the outputs of all heads. The number of heads per layer is fixed by the model's architecture, and methods that split a layer across devices, such as tensor parallelism, usually assign whole heads to each device.Attention Is All You NeedGlossary count. The server starts with --tensor-parallel-size <size> --gpu-memory-utilization 0.92.
  • Data parallelismdata parallelismA way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices.Hugging Face docsGlossary: One replica, unless BENCHMARK_VLLM_GPU_DP sets more.
  • Pipeline parallelismpipeline parallelismA way to split a model across several devices by assigning consecutive groups of its layers to each, so that each device holds only part of the weights. The output of one stage passes to the next, and the stages work on different requests or micro-batches at the same time to keep all devices busy. It needs less communication between devices than tensor parallelism.Hugging Face docsGlossary: On more than one GPU, the bitsandbytesbitsandbytesAn open-source Python library that quantizes the weights of PyTorch models to 8 or 4 bits, so that large models need less GPU memory. Hugging Face Transformers and vLLM can load models quantized with it, and its 4-bit format is also used to fine-tune large models on a single GPU.bitsandbytes on GitHubGlossary-quantized llama-70b starts with --pipeline-parallel-size <GPU count> instead, and without --gpu-memory-utilization--gpu-memory-utilizationThe vLLM option that sets the fraction of device memory, from 0 to 1, that a vLLM worker may use for the model weights, intermediate results, and key-value cache. On GPUs, it applies to the memory of each GPU; on the CPU backend of vLLM, each worker applies it to the total memory of its NUMA node, despite the option's name. The memory left after the weights and intermediate results goes to the key-value cache, so a higher value lets the server hold more concurrent sequences.vLLM engine argumentsGlossary. VLLM_GPU_MEMORY_UTILIZATION then affects only its memory fit check.

For example, on the UpCloud server with three NVIDIA L40SNVIDIA L40SA data center GPU of NVIDIA's Ada Lovelace generation with 48 GB of GDDR6 memory and compute capability 8.9. It is built for AI inference and training as well as graphics and video workloads.NVIDIAGlossary GPUs, smol-135m (9 attention heads) ran on all three GPUs, and qwen-0.5b (14 heads) and the models up to phi-4 on two (output). The llama-70b model ran with --pipeline-parallel-size 3 (log).

For llama-70b on more than one GPU, read pipeline_parallel in the output: tensor_parallel still shows the size that the head-count rule gives, and the log line of the parallel settings shows it too, although the server runs without tensor parallelism.

CPU Memory Fit​

Before it starts any server, the harness checks each model against the memory of the host, using the model's metadata from Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary. The check reads the host's memory, not the container's --memory limit, so the plan can count on memory that the container cannot use. The check works as follows:

  • A model runs when at least one tensor- and data-paralleldata parallelismA way to spread model training or inference across several devices by giving each a full copy of the model and splitting the data or requests among them. It adds throughput with each copy, but every device must hold all of the model's weights, unlike tensor and pipeline parallelism, which split one model across devices.Hugging Face docsGlossary layout fits: the weights and a minimum key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary of each vLLM worker must fit in the free memory of the NUMA node that the worker runs on.
  • When the serving data type is wider than the data type in the model's configuration, the check scales the size of the weights by the same ratio. For example, it doubles the weights of a bfloat16bfloat16A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively.WikipediaGlossary model that runs in float32float32A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory.WikipediaGlossary; see Data Type.
  • Before each workload, the same check runs with the workload's context length, and a workload whose key-value cache does not fit is skipped.
  • If the Hugging Face metadata cannot be read, the model is skipped, unless BENCHMARK_VLLM_ALLOW_ESTIMATED_METADATA is set.

On CPUs, vLLM reuses its --gpu-memory-utilization--gpu-memory-utilizationThe vLLM option that sets the fraction of device memory, from 0 to 1, that a vLLM worker may use for the model weights, intermediate results, and key-value cache. On GPUs, it applies to the memory of each GPU; on the CPU backend of vLLM, each worker applies it to the total memory of its NUMA node, despite the option's name. The memory left after the weights and intermediate results goes to the key-value cache, so a higher value lets the server hold more concurrent sequences.vLLM engine argumentsGlossary option, despite the name, to set the fraction of the NUMA node's total memory that each worker reserves for itself. What is left after the weights and intermediate results goes to the key-value cache, so a larger fraction lets a worker hold more requests at once. The harness splits the fraction among the workers that share a NUMA node, so that together they reserve at most 70% of the node's free memory, and caps each worker's fraction at 0.50. It leaves 0.8 GiB of each node for startup, and when several workers share a node, it lowers the 70% budget by a further 10%.

Set VLLM_CPU_GPU_MEMORY_UTILIZATION to override the fraction. Run the image with --plan-only to see the plan of each model on a host.

GPU Memory Fit​

Before it starts any server, the harness estimates how much memory each model needs on each GPU, without downloading the weights:

  • The weight size comes from the file metadata on Hugging FaceHugging FaceA company and online platform that hosts machine learning models, datasets, and demo apps in Git-based repositories, together known as the Hugging Face Hub. Programs download a model's files over HTTPS, optionally pinned to a specific commit, and gated models need an access token. Its Python libraries, such as Transformers, load and run the models.Hugging Face Hub docsGlossary, and the layer count, attention and key-value heads, hidden size, and data type from the model's config.json.
  • Each GPU needs the following, added together:
    • the weight size plus 10%, divided by the tensor-paralleltensor parallelismA way to split a model across several devices by dividing the weight matrices of each layer among them, so that every device computes part of every layer. The devices exchange partial results within each layer, which needs fast links between them, such as NVLink. The number of devices usually has to divide dimensions of the model, such as its attention head count.Hugging Face docsGlossary size times the pipeline-parallelpipeline parallelismA way to split a model across several devices by assigning consecutive groups of its layers to each, so that each device holds only part of the weights. The output of one stage passes to the next, and the stages work on different requests or micro-batches at the same time to keep all devices busy. It needs less communication between devices than tensor parallelism.Hugging Face docsGlossary size
    • the key-value cachekey-value cacheThe memory in which a transformer model keeps the attention keys and values of every token processed so far, so that each new token computes them only for itself. It grows with the context length and the number of concurrent sequences, and often limits how many requests an LLM server can batch at once. Serving engines such as vLLM manage it in fixed-size blocks.Hugging Face docsGlossary of one sequence of the full context length, rounded up to blocks of 16 tokens
    • 0.5 GiB
  • A model fits when every GPU it uses has that much memory, both within 92% of its total memory and within 95% of its free memory. Each GPU is checked on its own, not the total memory of all GPUs.

A model that does not fit is skipped, and the run continues with the next one. Before each workload, the same check runs with the workload's context length, and a workload that does not fit is skipped. If the Hugging Face metadata cannot be read, the model is skipped, unless BENCHMARK_VLLM_ALLOW_ESTIMATED_METADATA is set.

For example, on the g5.16xlarge, whose A10GNVIDIA A10GA data center GPU of NVIDIA's Ampere generation with 24 GB of GDDR6 memory and compute capability 8.6, a variant of the NVIDIA A10 made for AWS. It powers the EC2 G5 instances, which are used for machine learning inference and graphics workloads.AWSGlossary GPU has 22 GiB of memory, phi-4 and llama-70b were skipped for insufficient memory (log). Run the image with --plan-only to see the plan of each model on a host.

Data Type​

On CPUs, the harness runs the models in bfloat16bfloat16A 16-bit floating-point format designed for machine learning by Google Brain, a former AI research team at Google, now part of Google DeepMind. It keeps the 8-bit exponent of float32 but only 7 bits of its mantissa, so it covers the same range of values in half the memory, at a lower precision. Many CPUs, GPUs, and TPUs compute with it natively.WikipediaGlossary when /proc/cpuinfo lists the bf16 flag on Arm64Arm64The 64-bit execution state of the Arm architecture, also called AArch64, used by server CPUs such as AWS Graviton, Ampere Altra, and Google Axion. Linux distributions and Docker name it arm64 or aarch64.WikipediaGlossary or avx512_bf16 on AMD64AMD64The 64-bit version of the x86 instruction set architecture, introduced by AMD in 2003 and also implemented by Intel, which runs most cloud servers. Linux distributions and Docker name it amd64 or x86_64.WikipediaGlossary, and in float32float32A 32-bit floating-point format, also called single precision, defined by the IEEE 754 standard. It has a 1-bit sign, an 8-bit exponent, and a 23-bit mantissa, which gives about 7 decimal digits of precision and values up to about 3.4e38. It is the default number format of most machine learning frameworks, and formats such as float16 and bfloat16 trade some of its precision for less memory.WikipediaGlossary otherwise. It passes the data type to vllm serve as --dtype--dtypeThe vLLM option that sets the data type of the model weights and activations, such as bfloat16 or float32. With its default, auto, vLLM derives the data type from the model's configuration. A smaller data type needs less memory and bandwidth per weight, but only runs fast on hardware that supports it.vLLM engine argumentsGlossary, and records it in the dtype field of each line's tuning object. Set VLLM_CPU_DTYPE, such as -e VLLM_CPU_DTYPE=bfloat16, to use another data type; the memory check then uses the data type set.

Two server types can therefore serve the same model in different data types. An AMD64 CPU without AVX-512 has no avx512_bf16 flag, so it runs the models in float32, which takes 4 bytes per weight, twice as many as bfloat16. Such a host can fit fewer models than a host with the same memory that supports bfloat16.

The harness does not fall back to float16float16A 16-bit floating-point format, also called half precision, defined by the IEEE 754 standard. It has a 5-bit exponent and a 10-bit mantissa, so it takes half the memory of float32 but covers a much narrower range of values, up to 65,504. Machine learning frameworks use it to save memory and speed up computation.WikipediaGlossary: on AWS Graviton2 CPUs, which lack bf16, the vLLM server hangs with float16, as recorded in the harness.

On GPUs, the harness does not pass --dtype, so vLLM picks the data type.

Legacy Settings​

With BENCHMARK_VLLM_AUTOCONFIG=0, the harness uses static settings instead of the formulas in Load Settings:

SettingOn CPUsOn GPUs
Sweep size33
Maximum concurrency512512
GuideLLM worker processes1010
Throughput-stage ramp-upNoneNone
Seconds per stage45 + 12 * <parameters in billions>40 + 8 * <parameters in billions>
Request cap per stage25, set with GUIDELLM_MAX_REQUESTS_CPU120, set with GUIDELLM_MAX_REQUESTS

Legacy runs also use one vLLM server for all workloads, unless BENCHMARK_VLLM_PER_WORKLOAD_SERVER is set.

With autoconfig on or off, set GUIDELLM_PROFILES, or on CPUs GUIDELLM_CPU_PROFILES, to any of legacy, sync-throughput, or sync (all three do the same) to run one synchronous and one throughput stage instead of a sweep, with the throughput stage at a concurrency of GUIDELLM_THROUGHPUT_RATE (default: 8 on CPUs, 16 on GPUs).