Skip to main content

nvbandwidth

This benchmark measures how fast data moves between the host CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary and its NVIDIA GPUs, and between the GPUs themselves. It runs nvbandwidth v0.8, an open-source NVIDIA tool that times memory copies to measure bandwidth and memory access to measure latency. The copies run between the CPU and each GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary, between GPUs, and within the memory of a single GPU. The image runs every nvbandwidth test case in sequence, with the tool's default settings, on all GPUs passed to the container with --gpus all.

It reports one bandwidth matrix per test case in GB/s, or a latency matrix in nanoseconds, plus the matrix sum. The image adds no harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary and no score: the output is the unchanged nvbandwidth report.

Purpose​

Spare Cores Navigator publishes empirical performance measurements, such as raw CPU speed, memory bandwidth, and LLM inference speed, for more than 5,000 cloud server types. This benchmark adds measured raw GPU performance: how fast data moves between the CPU and the GPUs of a server type, and between its GPUs.

The results depend on the following properties of the host:

  • GPU model and count
  • links between the CPU and the GPUs and between GPUs, such as PCIePCIe: Peripheral Component Interconnect ExpressThe standard high-speed serial bus that connects expansion devices, such as GPUs, NVMe SSDs, and network cards, to a computer's CPU. A link has 1 to 16 lanes, and each generation roughly doubles the bandwidth per lane. A PCIe 4.0 link with 16 lanes transfers about 32 GB/s in each direction.WikipediaGlossary (Peripheral Component Interconnect Express) or NVLinkNVLinkA high-bandwidth, point-to-point interconnect by NVIDIA that links GPUs to each other, and in some systems to the CPU, directly rather than through PCIe. Between GPUs in the same server, it gives several times the bandwidth of a PCIe link, and NVSwitch chips connect many GPUs over NVLink at once.NVIDIAGlossary
  • host CPU and memory
  • NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary version, recorded by each run

Limitations​

This benchmark's results compare data movement on one server, not GPU compute speed or application performance. The main limitations are the following:

  • Copies Only: Every test case copies memory or measures memory access latency. Compute throughput, GPU memory capacity, and real workloads are not measured.
  • Single Server: nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary is built without its multinode mode, so only the CPU and GPUs of one host are measured, not links between servers or network throughput.
  • No Peer Access: The 17 test cases between GPUs need P2PP2P: Peer-to-PeerA model in which devices or computers exchange data directly with each other instead of through a central server or host. In GPU computing, it means copies and memory access between two GPUs over NVLink or PCIe, without staging the data in host memory; whether two GPUs support it depends on the hardware and the driver.CUDA programming guideGlossary (Peer-to-Peer) access between at least two GPUs. On a server with one GPU, or with GPUs that cannot access each other, they report "status": "Waived" and no matrix.
  • NVIDIA Only: nvbandwidth uses CUDACUDA: Compute Unified Device ArchitectureThe parallel computing platform and programming model of NVIDIA GPUs. It runs general-purpose code on the GPU, and most machine learning frameworks rely on its libraries, such as cuBLAS and cuDNN.NVIDIA docsGlossary (Compute Unified Device Architecture), so it needs NVIDIA GPUs. The image does not check for them; production skips servers without NVIDIA GPUs.
  • Fixed Settings: The image passes no options, so the upstream defaults apply. The bandwidth test cases use a 512 MiB copy buffer, four untimed warmup copies before the timed copies, and the median of three iterations per measurement. The two latency test cases always use a 2 MiB buffer and one timed pass, whatever the options. The image adds no warmup or repetitions of its own. See also Workload.
  • Host Driver: The host provides the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary, so its version varies between runs. The image fixes only the nvbandwidth version and the CUDA ToolkitCUDA ToolkitNVIDIA's development kit for building CUDA programs, with the nvcc compiler, headers, and libraries such as the CUDA runtime and cuBLAS. A program is built against one toolkit version and runs on any host whose NVIDIA driver supports that CUDA version, so the toolkit used for the build and the CUDA version a host reports can differ.NVIDIA docsGlossary 12.6.1 it is built with.

Usage​

Run the image via Docker, with the host's GPUs passed to the container:

docker run --rm --gpus all ghcr.io/sparecores/nvbandwidth:main

The host needs an NVIDIA GPU, the NVIDIA driverNVIDIA driverThe kernel module and libraries that let the operating system and programs use NVIDIA GPUs. Each driver version supports CUDA up to a certain version. Containers use the host's driver through the NVIDIA Container Toolkit, so it is installed on the host rather than in container images.NVIDIA docsGlossary, and the NVIDIA Container ToolkitNVIDIA Container ToolkitA set of NVIDIA libraries and tools that lets containers use the NVIDIA GPUs of the host. It configures container runtimes such as Docker to make the GPU devices and driver libraries available inside a container.NVIDIA docsGlossary.

The default command prints the nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary report as plain text. Production runs nvbandwidth -j instead, which prints JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary:

docker run --rm --gpus all ghcr.io/sparecores/nvbandwidth:main nvbandwidth -j

Arguments after the image name replace the default command, and still run under resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary, which samples resource usage during the run. Its data is not published.

Production also runs the container privileged and with host networking, with a 30-minute timeout. Production runs with nvbandwidth v0.8 took between 55 seconds and ~20 minutes, with a median of ~3.5 minutes; the slowest was on a server with eight GPUs.

Settings​

This benchmark has no settings of its own in environment variables. The image sets only TRACKER_QUIET=true, which silences the output of resource-tracker. To change a setting, pass nvbandwidth options after the image name, as in the JSON example under Usage. The most common options are the following:

OptionMeaningDefault
-jPrint JSON instead of plain textOff
-t <name or index>Run only the given test casesAll
-p <prefix>Run only the test cases whose names start with the prefixAll
-lList the test cases and exitOff
-b <MiB>Copy buffer size512
-i <count>Iterations per measurement3
-mReport the mean instead of the median of the iterationsOff
-sSkip the data check after each copyOff
-dTurn off automatic CPU affinityCPU affinityThe set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread binding or CPU pinning.Linux man pageGlossaryOff
-vVerbose outputOff

See the official nvbandwidth v0.8 documentation for all options and test cases.

Results​

With -j, nvbandwidth prints one JSON object to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary, with the following fields under its top-level nvbandwidth key:

  • CUDA Runtime Version and Driver Version record the CUDA version and the host's NVIDIA driver, for example 13000 (CUDA 13.0) and 580.126.09. Despite its name, CUDA Runtime Version holds the highest CUDA version that the host's driver supports, not the CUDA runtime in the image, which is built with CUDA ToolkitCUDA ToolkitNVIDIA's development kit for building CUDA programs, with the nvcc compiler, headers, and libraries such as the CUDA runtime and cuBLAS. A program is built against one toolkit version and runs on any host whose NVIDIA driver supports that CUDA version, so the toolkit used for the build and the CUDA version a host reports can differ.NVIDIA docsGlossary 12.6.1.
  • GPU Device list names each GPU, with its index, name, bus address, and hostname.
  • version is the nvbandwidth version (v0.8). The git_version field is empty, because the image builds nvbandwidth from a release archive.
  • testcases[] has one entry per test case, with its name and status.

A test case that ran has "status": "Passed" and the following fields:

  • bandwidth_description says what the matrix holds and in which unit, for example memcpy CE CPU(row) -> GPU(column) bandwidth (GB/s).
  • bandwidth_matrix is a two-dimensional array of strings, with rows and columns as the description says. Pairs that were not measured hold N/A.
  • sum is the sum of the matrix cells, as a number.

The four device_to_device_bidirectional_memcpy_* test cases are the exception: their bandwidth_matrix stacks three GPU-by-GPU matrices, one for each direction and then their total, where each cell adds up both directions. Their sum and bandwidth_description cover only the last, total matrix.

The 17 test cases between GPUs need P2P access between at least two GPUs. Without it, on a server with one GPU or with GPUs that cannot access each other, they report "status": "Waived" and have no matrix. In production, 16 of 27 runs with nvbandwidth v0.8 on servers with two to four GPUs waived all 17.

For example, on an alicloud/ecs.gn5-c4g1.xlarge with one Tesla P100 GPU, host_to_device_memcpy_ce measured 11.06 GB/s, host_device_latency_sm 799 ns, and device_local_copy 268.75 GB/s, while all 17 test cases between GPUs were waived (production output).

On a fatal error, such as a GPU that CUDA cannot initialize or a failed data check, nvbandwidth adds an error field to the JSON, prints it with the test cases that ran before the error, and exits with code 1. The test case that was running keeps "status": "Running". A failed data check also prints Invalid value when checking the pattern at ... before the JSON.

The Navigator stores the following metrics from each run, computed only from test cases with "status": "Passed", and skips runs with an error field. Its Web UI shows them in the GPU Bandwidth and Latency card of each server type's page. The table uses the following terms:

  • CECE: Copy EngineA DMA (Direct Memory Access) engine of an NVIDIA GPU that copies data between GPU memory, host memory, and the memory of other GPUs without using the GPU's compute cores. A GPU has one or more copy engines, which CUDA uses for memory copies, so data transfers can run at the same time as compute kernels.CUDA programming guideGlossary (Copy Engine) and SMSM: Streaming MultiprocessorThe basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number.CUDA programming guideGlossary (Streaming Multiprocessor) name the copy method; see Workload.
  • P2P means direct copies between GPUs.
  • "Averaged" means the sum divided by the number of cells that are not N/A. For a stacked bidirectional matrix, only the cells of the total matrix count, and the result is also halved, which gives the rate per direction.
MetricMeaningUnitComputed from
nvbandwidth:all:host_to_gpuTotal rate when the CPU copies to all GPUs at once (CE)GB/shost_to_all_memcpy_ce sum
nvbandwidth:all:gpu_to_hostTotal rate when all GPUs copy to the CPU at once (CE)GB/sall_to_host_memcpy_ce sum
nvbandwidth:all:duplexPer-direction rate when copying both ways at once, the lower of the two directions (CE)GB/sLower sum of host_to_all_bidirectional_memcpy_ce and all_to_host_bidirectional_memcpy_ce
nvbandwidth:slot:host_to_gpuAverage PCIe slot speed per GPU, CPU to GPU (CE)GB/shost_to_device_memcpy_ce, averaged
nvbandwidth:slot:gpu_to_hostAverage PCIe slot speed per GPU, GPU to CPU (CE)GB/sdevice_to_host_memcpy_ce, averaged
nvbandwidth:slot:local_copyAverage copy speed within each GPU's own memory (CE)GB/sdevice_local_copy, averaged
nvbandwidth:slot:latencyAverage CPU-GPU round-trip time (SM); lower is betternshost_device_latency_sm, averaged
nvbandwidth:p2p:singleWrite speed between two GPUs with P2P copies (CE)GB/sdevice_to_device_memcpy_write_ce, averaged
nvbandwidth:p2p:duplexPer-direction speed between two GPUs copying both ways at once (CE)GB/sdevice_to_device_bidirectional_memcpy_write_ce, averaged
nvbandwidth:p2p:latencyLatency between GPU pairs with P2P access (SM); lower is betternsdevice_to_device_latency_sm, averaged
nvbandwidth:p2p:gatherRate into one GPU when all other GPUs write to it at once (CE)GB/sall_to_one_write_ce, averaged
nvbandwidth:efficiency:sm_ce_ratioSM copy speed relative to CE copy speed, capped at 1Ratiohost_to_device_memcpy_sm sum divided by host_to_device_memcpy_ce sum

The card also shows the following, which each stored result records:

  • driver version
  • CUDA version that the driver supports
  • nvbandwidth version
  • GPU count
  • whether P2P is supported, that is, whether any device_to_device_* test case passed

Workload​

This benchmark runs one workload: the image's default command, nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary with no options, which runs all 35 test cases in sequence. The Dockerfile builds nvbandwidth v0.8 from NVIDIA's release archive with CUDA ToolkitCUDA ToolkitNVIDIA's development kit for building CUDA programs, with the nvcc compiler, headers, and libraries such as the CUDA runtime and cuBLAS. A program is built against one toolkit version and runs on any host whose NVIDIA driver supports that CUDA version, so the toolkit used for the build and the CUDA version a host reports can differ.NVIDIA docsGlossary 12.6.1.

Each nvbandwidth test casenvbandwidth test caseOne named measurement of the nvbandwidth tool, such as host_to_device_memcpy_ce, defined by its copy direction, the GPUs taking part, and its copy method, given by the _ce or _sm suffix. A test case between GPUs is waived rather than run when no two GPUs of the system have P2P (Peer-to-Peer) access to each other, as on a system with one GPU. The nvbandwidth -l command lists the test cases of a build.Official documentationGlossary uses one of the following copy methods, named by its suffix:

  • CE: _ce test cases copy with the GPU's copy engines, through CUDA's memcpy functions.
  • SM: _sm test cases copy with a kernel that runs on the GPU's streaming multiprocessors, with 512 threads per block. The copy size is rounded down so that it divides evenly across the SMs.

The 16 CE and 16 SM test cases cover the following copy directions:

  • host_to_device_* and device_to_host_*: between the CPU and each GPU, one GPU at a time.
  • host_to_all_* and all_to_host_*: between the CPU and all GPUs at once.
  • *_bidirectional_*: the same copies in both directions at once.
  • device_to_device_memcpy_read_* and device_to_device_memcpy_write_*: between each pair of GPUs.
  • all_to_one_* and one_to_all_*: from all GPUs to one, and from one GPU to all others, with a read and a write variant each.

Three more test cases complete the set: host_device_latency_sm and device_to_device_latency_sm measure memory access latency between the CPU and each GPU and between GPUs, and device_local_copy measures copies within each GPU's own memory.

Each bandwidth measurement times a fixed number of copies between start and stop events, repeats three times, and reports the median. Four warmup copies run before the timed copies, outside the timed interval. nvbandwidth queues all copies before releasing them, so the time spent queuing them is not measured. It checks the copied data after each copy and sets CPU affinityCPU affinityThe set of CPU cores that a thread or process may run on, which the operating system's scheduler keeps to when it places the thread. It includes every core by default; restricting it to chosen cores is called thread binding or CPU pinning.Linux man pageGlossary automatically.

The latency test cases instead follow a chain of pointers through a 2 MiB buffer, so each read waits for the one before it. They time one million such reads in one pass, without warmup, and report the average time per read. The -b, -i, and -m options do not apply to them.

Design History​

This benchmark has no harness of its own: the image runs the upstream nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary binary, so the test cases and their settings are NVIDIA's. The image first shipped nvbandwidth v0.5 and moved to v0.8 in 2026. Production records the version of each run, and many server types measured before the update still have v0.5 results.

References​

See the full list of references used in this documentation.