llm_speed
LLM inference speed
LLM inference speed for prompt processing
Running llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary from llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary using various quantizedquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary model files to measure the speed of processing 16 to 16k tokenstokenThe unit of text that an LLM reads and writes, as defined by the model's tokenizer, such as a word, part of a word, or a punctuation mark. Prompt and output lengths are counted in tokens, and LLM inference speed is usually measured in tokens per second.Glossary.
Benchmark details
- Benchmark ID
llm_speed:prompt_processing- Unit of measurement
- tokens/second (t/s)
- Server types evaluated
- 3,453
- Total measurements
- 63,925
- Range of measurements
- 0.539 – 102.9K t/s
- Better
- ↑ Higher is better
Configuration options
| Field | Description | Example values |
|---|---|---|
model | Name of the model file used. | Llama-3.3-70B-Instruct-Q4_K_M.gguf SmolLM-135M.Q4_K_M.gguf gemma-2b.Q4_K_M.gguf llama-7b.Q4_K_M.gguf phi-4-q4.gguf qwen1_5-0_5b-chat-q4_k_m.gguf |
tokens | Number of tokens processed in one run. | 16 128 512 1024 4096 16384 |
Throughput depends on model size and hardware. On CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary, smaller models often top out around 96 vCPUsvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary; larger models hit memory limits sooner. GPUs are much faster than CPUs for this workload. llama.cpp only uses multiple GPUs when one card cannot hold the model weights. Extra GPUs are for fitting the model, not for serving more requests in parallel. These runs target single-user, low-concurrency inference. For high concurrency, use the vLLMvLLM: virtual Large Language ModelAn open-source inference and serving engine for LLMs, started at UC Berkeley in 2023. Its PagedAttention technique manages the key-value cache in fixed-size blocks, as an OS pages virtual memory, and it batches concurrent requests continuously, so one server can answer many requests at once. It runs on GPUs and CPUs and serves models through an OpenAI-compatible HTTP API.vLLM docsvLLM on GitHubOrigin of the nameGlossary benchmarks.
Score distribution (63,925 measurements)
LLM inference speed for text generation
Running llama-benchllama-benchThe benchmark tool of llama.cpp. It measures the speed of prompt processing and text generation in tokens per second, for given models, token counts, and settings. Each test is repeated several times, and the tool reports the average and standard deviation, as a table or in machine-readable formats such as JSON Lines.llama-bench READMEGlossary from llama.cppllama.cppAn open-source C/C++ library and set of tools for LLM inference, started by Georgi Gerganov in 2023 and built on the ggml tensor library. It runs models in the GGUF format, usually quantized, on CPUs of many architectures, with code optimized for their SIMD instruction sets, and on GPUs through backends such as CUDA, Metal, and Vulkan. Its tools include a command-line client, an HTTP server, and a benchmark tool, llama-bench.llama.cpp on GitHubGlossary using various quantizedquantizationStoring the weights of a neural network with fewer bits per value than its training format, such as 4 or 8 bits instead of 16, usually with a shared scale factor per block of weights. It shrinks the model's file size and memory use and speeds up inference, especially where memory bandwidth is the limit, at the cost of some accuracy.Hugging Face docsGlossary model files to measure the speed of generating 16 to 4k tokens.
Benchmark details
- Benchmark ID
llm_speed:text_generation- Unit of measurement
- tokens/second (t/s)
- Server types evaluated
- 3,450
- Total measurements
- 48,591
- Range of measurements
- 0.377 – 698.3 t/s
- Better
- ↑ Higher is better
Configuration options
| Field | Description | Example values |
|---|---|---|
model | Name of the model file used. | Llama-3.3-70B-Instruct-Q4_K_M.gguf SmolLM-135M.Q4_K_M.gguf gemma-2b.Q4_K_M.gguf llama-7b.Q4_K_M.gguf phi-4-q4.gguf qwen1_5-0_5b-chat-q4_k_m.gguf |
tokens | Number of tokens processed in one run. | 16 128 512 1024 4096 |
Throughput depends on model size and hardware. On CPU, smaller models often top out around 96 vCPUs; larger models hit memory limits sooner. GPUs are much faster than CPUs for this workload. llama.cpp only uses multiple GPUs when one card cannot hold the model weights. Extra GPUs are for fitting the model, not for serving more requests in parallel. These runs target single-user, low-concurrency inference. For high concurrency, use the vLLM benchmarks.
Score distribution (48,591 measurements)