# llm_speed

> LLM inference speed benchmark workloads: LLM inference speed for prompt processing, LLM inference speed for text generation.

LLM inference speed

## LLM inference speed for prompt processing {#llm_speed-prompt_processing}

Running llama-bench from llama.cpp using various quantized model files to measure the speed of processing 16 to 16k tokens.

Benchmark details<dl><dt>Benchmark ID</dt><dd><code>llm_speed:prompt_processing</code></dd><dt>Unit of measurement</dt><dd>tokens/second (t/s)</dd><dt>Server types evaluated</dt><dd>3,453</dd><dt>Total measurements</dt><dd>63,925</dd><dt>Range of measurements</dt><dd>0.539 – 102.9K <small>t/s</small></dd><dt>Better</dt><dd>↑ Higher is better</dd></dl>

Configuration options

| Field | Description | Example values |
| --- | --- | --- |
| `model` | Name of the model file used. | `Llama-3.3-70B-Instruct-Q4_K_M.gguf` `SmolLM-135M.Q4_K_M.gguf` `gemma-2b.Q4_K_M.gguf` `llama-7b.Q4_K_M.gguf` `phi-4-q4.gguf` `qwen1_5-0_5b-chat-q4_k_m.gguf` |
| `tokens` | Number of tokens processed in one run. | `16` `128` `512` `1024` `4096` `16384` |

:::warning

Throughput depends on model size and hardware. On CPU, smaller models often top out around 96 vCPUs; larger models hit memory limits sooner. GPUs are much faster than CPUs for this workload. llama.cpp only uses multiple GPUs when one card cannot hold the model weights. Extra GPUs are for fitting the model, not for serving more requests in parallel. These runs target single-user, low-concurrency inference. For high concurrency, use the vLLM benchmarks.

:::

Score distribution <small>(63,925 measurements)</small>

![Score distribution of LLM inference speed for prompt processing](/benchmarks/histograms/llm_speed-prompt_processing.svg)

## LLM inference speed for text generation {#llm_speed-text_generation}

Running llama-bench from llama.cpp using various quantized model files to measure the speed of generating 16 to 4k tokens.

Benchmark details<dl><dt>Benchmark ID</dt><dd><code>llm_speed:text_generation</code></dd><dt>Unit of measurement</dt><dd>tokens/second (t/s)</dd><dt>Server types evaluated</dt><dd>3,450</dd><dt>Total measurements</dt><dd>48,591</dd><dt>Range of measurements</dt><dd>0.377 – 698.3 <small>t/s</small></dd><dt>Better</dt><dd>↑ Higher is better</dd></dl>

Configuration options

| Field | Description | Example values |
| --- | --- | --- |
| `model` | Name of the model file used. | `Llama-3.3-70B-Instruct-Q4_K_M.gguf` `SmolLM-135M.Q4_K_M.gguf` `gemma-2b.Q4_K_M.gguf` `llama-7b.Q4_K_M.gguf` `phi-4-q4.gguf` `qwen1_5-0_5b-chat-q4_k_m.gguf` |
| `tokens` | Number of tokens processed in one run. | `16` `128` `512` `1024` `4096` |

:::warning

Throughput depends on model size and hardware. On CPU, smaller models often top out around 96 vCPUs; larger models hit memory limits sooner. GPUs are much faster than CPUs for this workload. llama.cpp only uses multiple GPUs when one card cannot hold the model weights. Extra GPUs are for fitting the model, not for serving more requests in parallel. These runs target single-user, low-concurrency inference. For high concurrency, use the vLLM benchmarks.

:::

Score distribution <small>(48,591 measurements)</small>

![Score distribution of LLM inference speed for text generation](/benchmarks/histograms/llm_speed-text_generation.svg)
