nvbandwidth
This benchmark measures how fast data moves between the host CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary and its
NVIDIA GPUs, and between the GPUs themselves. It runs nvbandwidth v0.8, an
open-source NVIDIA tool that times memory copies to measure bandwidth and
memory access to measure latency. The copies run between the CPU and each
GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary, between GPUs, and within the memory of a single GPU. The image runs
every nvbandwidth test case in sequence, with the tool's default settings,
on all GPUs passed to the container with --gpus all.
It reports one bandwidth matrix per test case in GB/s, or a latency matrix in
nanoseconds, plus the matrix sum. The image adds no harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary and no score: the
output is the unchanged nvbandwidth report.
Purpose
Spare Cores Navigator publishes empirical performance measurements, such as raw CPU speed, memory bandwidth, and LLM inference speed, for more than 5,000 cloud server types. This benchmark adds measured raw GPU performance: how fast data moves between the CPU and the GPUs of a server type, and between its GPUs.
The results depend on the following properties of the host:
- GPU model and count
- links between the CPU and the GPUs and between GPUs, such as PCIePCIe: Peripheral Component Interconnect ExpressThe standard high-speed serial bus that connects expansion devices, such as GPUs, NVMe SSDs, and network cards, to a computer's CPU. A link has 1 to 16 lanes, and each generation roughly doubles the bandwidth per lane. A PCIe 4.0 link with 16 lanes transfers about 32 GB/s in each direction.WikipediaGlossary (Peripheral Component Interconnect Express) or NVLinkNVLinkA high-bandwidth, point-to-point interconnect by NVIDIA that links GPUs to each other, and in some systems to the CPU, directly rather than through PCIe. Between GPUs in the same server, it gives several times the bandwidth of a PCIe link, and NVSwitch chips connect many GPUs over NVLink at once.NVIDIAGlossary
- host CPU and memory
- NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary version, recorded by each run
Limitations
This benchmark's results compare data movement on one server, not GPU compute speed or application performance. The main limitations are the following:
- Copies Only: Every test case copies memory or measures memory access latency. Compute throughput, GPU memory capacity, and real workloads are not measured.
- Single Server:
nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary is built without its multinode mode, so only the CPU and GPUs of one host are measured, not links between servers or network throughput. - No Peer Access: The 17 test cases between GPUs need P2PP2P: Peer-to-PeerA model in which devices or computers exchange data directly with each other instead of through a central server or host. In GPU computing, it means copies and memory access between two GPUs over NVLink or PCIe, without staging the data in host memory; whether two GPUs support it depends on the hardware and the driver.CUDA programming guideGlossary
(Peer-to-Peer) access between at least two GPUs. On a server with one GPU,
or with GPUs that cannot access each other, they report
"status": "Waived"and no matrix. - NVIDIA Only:
nvbandwidthuses CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture), so it needs NVIDIA GPUs. The image does not check for them; production skips servers without NVIDIA GPUs. - Fixed Settings: The image passes no options, so the upstream defaults apply. The bandwidth test cases use a 512 MiB copy buffer, four untimed warmup copies before the timed copies, and the median of three iterations per measurement. The two latency test cases always use a 2 MiB buffer and one timed pass, whatever the options. The image adds no warmup or repetitions of its own. See also Workload.
- Host Driver: The host provides the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary, so its version varies
between runs. The image fixes only the
nvbandwidthversion and the CUDA ToolkitCUDA ToolkitNVIDIA's development kit for building CUDA programs, with the nvcc compiler, headers, and libraries such as the CUDA runtime and cuBLAS. A program is built against one toolkit version and runs on any host whose NVIDIA driver supports that CUDA version, so the toolkit used for the build and the CUDA version a host reports can differ.NVIDIA docsGlossary 12.6.1 it is built with.
Usage
Run the image via Docker, with the host's GPUs passed to the container:
docker run --rm --gpus all ghcr.io/sparecores/nvbandwidth:main
The host needs an NVIDIA GPU, the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary, and the NVIDIA Container ToolkitNVIDIA Container ToolkitA set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container.NVIDIA docsGlossary.
The default command prints the nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary report as plain text. Production
runs nvbandwidth -j instead, which prints JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary:
docker run --rm --gpus all ghcr.io/sparecores/nvbandwidth:main nvbandwidth -j
Arguments after the image name replace the default command, and still run
under resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary, which samples resource usage during the run. Its
data is not published.
Production also runs the container privileged and with host networking, with a
30-minute timeout. Production runs with nvbandwidth v0.8 took between 55
seconds and ~20 minutes, with a median of ~3.5 minutes; the slowest
was on a server with eight GPUs.
Settings
This benchmark has no settings of its own in environment variables. The image
sets only TRACKER_QUIET=true, which silences the output of
resource-tracker. To change a setting, pass nvbandwidth options after the
image name, as in the JSON example under Usage. The most common
options are the following:
| Option | Meaning | Default |
|---|---|---|
-j | Print JSON instead of plain text | Off |
-t <name or index> | Run only the given test cases | All |
-p <prefix> | Run only the test cases whose names start with the prefix | All |
-l | List the test cases and exit | Off |
-b <MiB> | Copy buffer size | 512 |
-i <count> | Iterations per measurement | 3 |
-m | Report the mean instead of the median of the iterations | Off |
-s | Skip the data check after each copy | Off |
-d | Turn off automatic CPU affinityCPU affinityThe set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread binding or CPU pinning.Linux man pageGlossary | Off |
-v | Verbose output | Off |
See the
official nvbandwidth v0.8 documentation
for all options and test cases.
Results
With -j, nvbandwidth prints one JSON object to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary, with the following
fields under its top-level nvbandwidth key:
CUDA Runtime VersionandDriver Versionrecord the CUDA version and the host's NVIDIA driver, for example13000(CUDA 13.0) and580.126.09. Despite its name,CUDA Runtime Versionholds the highest CUDA version that the host's driver supports, not the CUDA runtime in the image, which is built with CUDA ToolkitCUDA ToolkitNVIDIA's development kit for building CUDA programs, with the nvcc compiler, headers, and libraries such as the CUDA runtime and cuBLAS. A program is built against one toolkit version and runs on any host whose NVIDIA driver supports that CUDA version, so the toolkit used for the build and the CUDA version a host reports can differ.NVIDIA docsGlossary 12.6.1.GPU Device listnames each GPU, with its index, name, bus address, and hostname.versionis thenvbandwidthversion (v0.8). Thegit_versionfield is empty, because the image buildsnvbandwidthfrom a release archive.testcases[]has one entry per test case, with itsnameandstatus.
A test case that ran has "status": "Passed" and the following fields:
bandwidth_descriptionsays what the matrix holds and in which unit, for examplememcpy CE CPU(row) -> GPU(column) bandwidth (GB/s).bandwidth_matrixis a two-dimensional array of strings, with rows and columns as the description says. Pairs that were not measured holdN/A.sumis the sum of the matrix cells, as a number.
The four device_to_device_bidirectional_memcpy_* test cases are the
exception: their bandwidth_matrix stacks three GPU-by-GPU matrices, one for
each direction and then their total, where each cell adds up both
directions. Their sum and bandwidth_description cover only the last, total
matrix.
The 17 test cases between GPUs need P2P access between at least two GPUs.
Without it, on a server with one GPU or with GPUs that cannot access each
other, they report "status": "Waived" and have no matrix. In production,
16 of 27 runs with nvbandwidth v0.8 on servers with two to four GPUs waived
all 17.
For example, on an
alicloud/ecs.gn5-c4g1.xlarge
with one Tesla P100 GPU, host_to_device_memcpy_ce measured 11.06 GB/s,
host_device_latency_sm 799 ns, and device_local_copy 268.75 GB/s, while all
17 test cases between GPUs were waived
(production output).
On a fatal error, such as a GPU that CUDA cannot initialize or a failed data
check, nvbandwidth adds an error field to the JSON, prints it with the test
cases that ran before the error, and exits with code 1. The test case that was
running keeps "status": "Running". A failed data check also prints
Invalid value when checking the pattern at ... before the JSON.
The Navigator stores the following metrics from each run, computed only from
test cases with "status": "Passed", and skips runs with an error field.
Its Web UI shows them in the GPU Bandwidth and Latency card of each server
type's page. The table uses the following terms:
- CECE: Copy EngineA DMA (Direct Memory Access) engine of an NVIDIA GPU that copies data between GPU memory, host memory, and the memory of other GPUs without using the GPU's compute cores. A GPU has one or more copy engines, which CUDA uses for memory copies, so data transfers can run at the same time as compute kernels.CUDA programming guideGlossary (Copy Engine) and SMSM: Streaming MultiprocessorThe basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number.CUDA programming guideGlossary (Streaming Multiprocessor) name the copy method; see Workload.
- P2P means direct copies between GPUs.
- "Averaged" means the
sumdivided by the number of cells that are notN/A. For a stacked bidirectional matrix, only the cells of the total matrix count, and the result is also halved, which gives the rate per direction.
| Metric | Meaning | Unit | Computed from |
|---|---|---|---|
nvbandwidth:all:host_to_gpu | Total rate when the CPU copies to all GPUs at once (CE) | GB/s | host_to_all_memcpy_ce sum |
nvbandwidth:all:gpu_to_host | Total rate when all GPUs copy to the CPU at once (CE) | GB/s | all_to_host_memcpy_ce sum |
nvbandwidth:all:duplex | Per-direction rate when copying both ways at once, the lower of the two directions (CE) | GB/s | Lower sum of host_to_all_bidirectional_memcpy_ce and all_to_host_bidirectional_memcpy_ce |
nvbandwidth:slot:host_to_gpu | Average PCIe slot speed per GPU, CPU to GPU (CE) | GB/s | host_to_device_memcpy_ce, averaged |
nvbandwidth:slot:gpu_to_host | Average PCIe slot speed per GPU, GPU to CPU (CE) | GB/s | device_to_host_memcpy_ce, averaged |
nvbandwidth:slot:local_copy | Average copy speed within each GPU's own memory (CE) | GB/s | device_local_copy, averaged |
nvbandwidth:slot:latency | Average CPU-GPU round-trip time (SM); lower is better | ns | host_device_latency_sm, averaged |
nvbandwidth:p2p:single | Write speed between two GPUs with P2P copies (CE) | GB/s | device_to_device_memcpy_write_ce, averaged |
nvbandwidth:p2p:duplex | Per-direction speed between two GPUs copying both ways at once (CE) | GB/s | device_to_device_bidirectional_memcpy_write_ce, averaged |
nvbandwidth:p2p:latency | Latency between GPU pairs with P2P access (SM); lower is better | ns | device_to_device_latency_sm, averaged |
nvbandwidth:p2p:gather | Rate into one GPU when all other GPUs write to it at once (CE) | GB/s | all_to_one_write_ce, averaged |
nvbandwidth:efficiency:sm_ce_ratio | SM copy speed relative to CE copy speed, capped at 1 | Ratio | host_to_device_memcpy_sm sum divided by host_to_device_memcpy_ce sum |
The card also shows the following, which each stored result records:
- driver version
- CUDA version that the driver supports
nvbandwidthversion- GPU count
- whether P2P is supported, that is, whether any
device_to_device_*test case passed
Workload
This benchmark runs one workload: the image's default command,
nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary with no options, which runs all 35 test cases in sequence. The
Dockerfile builds nvbandwidth v0.8 from NVIDIA's release
archive with CUDA ToolkitCUDA ToolkitNVIDIA's development kit for building CUDA programs, with the nvcc compiler, headers, and libraries such as the CUDA runtime and cuBLAS. A program is built against one toolkit version and runs on any host whose NVIDIA driver supports that CUDA version, so the toolkit used for the build and the CUDA version a host reports can differ.NVIDIA docsGlossary 12.6.1.
Each nvbandwidth test casenvbandwidth test caseOne named measurement of the nvbandwidth tool, such as host_to_device_memcpy_ce, defined by its copy direction, the GPUs taking part, and its copy method, given by the _ce or _sm suffix. A test case between GPUs is waived rather than run when no two GPUs of the system have P2P (Peer-to-Peer) access to each other, as on a system with one GPU. The nvbandwidth -l command lists the test cases of a build.Official documentationGlossary uses one of the following copy methods, named by its
suffix:
- CE:
_cetest cases copy with the GPU's copy engines, through CUDA'smemcpyfunctions. - SM:
_smtest cases copy with a kernel that runs on the GPU's streaming multiprocessors, with 512 threads per block. The copy size is rounded down so that it divides evenly across the SMs.
The 16 CE and 16 SM test cases cover the following copy directions:
host_to_device_*anddevice_to_host_*: between the CPU and each GPU, one GPU at a time.host_to_all_*andall_to_host_*: between the CPU and all GPUs at once.*_bidirectional_*: the same copies in both directions at once.device_to_device_memcpy_read_*anddevice_to_device_memcpy_write_*: between each pair of GPUs.all_to_one_*andone_to_all_*: from all GPUs to one, and from one GPU to all others, with a read and a write variant each.
Three more test cases complete the set: host_device_latency_sm and
device_to_device_latency_sm measure memory access latency between the CPU
and each GPU and between GPUs, and device_local_copy measures copies within
each GPU's own memory.
Each bandwidth measurement times a fixed number of copies between start and
stop events, repeats three times, and reports the median. Four warmup copies
run before the timed copies, outside the timed interval. nvbandwidth queues
all copies before releasing them, so the time spent queuing them is not
measured. It checks the copied data after each copy and sets CPU affinityCPU affinityThe set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread binding or CPU pinning.Linux man pageGlossary
automatically.
The latency test cases instead follow a chain of pointers through a 2 MiB
buffer, so each read waits for the one before it. They time one million such
reads in one pass, without warmup, and report the average time per read. The
-b, -i, and -m options do not apply to them.
Design History
This benchmark has no harness of its own: the image runs the upstream
nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary binary, so the test cases and their settings are NVIDIA's. The
image first shipped nvbandwidth v0.5 and moved to v0.8 in 2026. Production
records the version of each run, and many server types measured before the
update still have v0.5 results.
References
See the full list of references used in this documentation.