Skip to main content

nvbandwidth

GPU bandwidth, GPU latency

Node baseline directional bandwidth (nvbandwidth)​

Conservative per-direction bandwidth (GB/s) under simultaneous bi-directional saturation utilizing CECE: Copy EngineA DMA (Direct Memory Access) engine of an NVIDIA GPU that copies data between GPU memory, host memory, and the memory of other GPUs without using the GPU's compute cores. A GPU has one or more copy engines, which CUDA uses for memory copies, so data transfers can run at the same time as compute kernels.CUDA programming guideGlossary (Copy Engine).

Benchmark details

Benchmark ID
nvbandwidth:all:duplex
Unit of measurement
GB/s
Server types evaluated
180
Total measurements
180
Range of measurements
5.3 – 178.6 GB/s
Better
↑ Higher is better

Score distribution (180 measurements)

Score distribution of Node baseline directional bandwidth (nvbandwidth)

Node GPU-to-host bandwidth (nvbandwidth)​

Total node egress rate (GB/s) when ALL GPUs stream to CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary concurrently utilizing CE (Copy Engine).

Benchmark details

Benchmark ID
nvbandwidth:all:gpu_to_host
Unit of measurement
GB/s
Server types evaluated
180
Total measurements
180
Range of measurements
6.6 – 229.3 GB/s
Better
↑ Higher is better

Score distribution (180 measurements)

Score distribution of Node GPU-to-host bandwidth (nvbandwidth)

Node host-to-GPU bandwidth (nvbandwidth)​

Total node ingest rate (GB/s) when CPU streams to ALL GPUs concurrently utilizing CE (Copy Engine).

Benchmark details

Benchmark ID
nvbandwidth:all:host_to_gpu
Unit of measurement
GB/s
Server types evaluated
180
Total measurements
180
Range of measurements
6.2 – 228.8 GB/s
Better
↑ Higher is better

Score distribution (180 measurements)

Score distribution of Node host-to-GPU bandwidth (nvbandwidth)

SM/CE copy efficiency (nvbandwidth)​

Ratio (between 0.0 and 1.0) showing compute-core copy efficiency vs. raw DMA Copy Engine capacity, or in other terms, the ratio between CE (Copy Engine) and Streaming Multiprocessor (SMSM: Streaming MultiprocessorThe basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number.CUDA programming guideGlossary).

Benchmark details

Benchmark ID
nvbandwidth:efficiency:sm_ce_ratio
Unit of measurement
ratio
Server types evaluated
180
Total measurements
180
Range of measurements
0.0717 – 1 ratio
Better
↑ Higher is better

Score distribution (180 measurements)

Score distribution of SM/CE copy efficiency (nvbandwidth)

GPU-to-GPU P2P duplex bandwidth (nvbandwidth)​

Per-direction NVLink/P2P lane speed (GB/s) during simultaneous transfers in both directions between two GPUs utilizing CE (Copy Engine).

Benchmark details

Benchmark ID
nvbandwidth:p2p:duplex
Unit of measurement
GB/s
Server types evaluated
33
Total measurements
33
Range of measurements
4.9 – 770.4 GB/s
Better
↑ Higher is better

Score distribution (33 measurements)

Score distribution of GPU-to-GPU P2P duplex bandwidth (nvbandwidth)

GPU-to-GPU P2P gather bandwidth (nvbandwidth)​

Intake bandwidth (GB/s) on a single target GPUGPU: Graphics Processing UnitA processor with thousands of simple cores that run the same operation on many data elements in parallel. It was built to render graphics, and is now also used for general-purpose parallel work such as machine learning training and inference, scientific simulation, and video encoding. Cloud providers rent GPUs attached to virtual machines, with their own dedicated GPU memory.WikipediaGlossary when all peer GPUs send to it at once utilizing CE (Copy Engine).

Benchmark details

Benchmark ID
nvbandwidth:p2p:gather
Unit of measurement
GB/s
Server types evaluated
33
Total measurements
33
Range of measurements
9.4 – 795.3 GB/s
Better
↑ Higher is better

Score distribution (33 measurements)

Score distribution of GPU-to-GPU P2P gather bandwidth (nvbandwidth)

GPU-to-GPU P2P latency (nvbandwidth)​

Direct transit latency (ns) between GPU pairs over NVLink/P2P utilizing SMSM: Streaming MultiprocessorThe basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number.CUDA programming guideGlossary (Streaming Multiprocessor).

Benchmark details

Benchmark ID
nvbandwidth:p2p:latency
Unit of measurement
ns
Server types evaluated
33
Total measurements
33
Range of measurements
394.7 – 1.6K ns
Better
↓ Lower is better

Score distribution (33 measurements)

Score distribution of GPU-to-GPU P2P latency (nvbandwidth)

GPU-to-GPU P2P bandwidth (nvbandwidth)​

Single NVLink/P2P lane write speed (GB/s) between any two GPUs utilizing CE (Copy Engine).

Benchmark details

Benchmark ID
nvbandwidth:p2p:single
Unit of measurement
GB/s
Server types evaluated
33
Total measurements
33
Range of measurements
9.9 – 774.7 GB/s
Better
↑ Higher is better

Score distribution (33 measurements)

Score distribution of GPU-to-GPU P2P bandwidth (nvbandwidth)

Per-GPU GPU-to-host bandwidth (nvbandwidth)​

Average physical PCIe slot egress speed (GB/s) per single GPU utilizing CE (Copy Engine).

Benchmark details

Benchmark ID
nvbandwidth:slot:gpu_to_host
Unit of measurement
GB/s
Server types evaluated
180
Total measurements
180
Range of measurements
6.6 – 57.4 GB/s
Better
↑ Higher is better

Score distribution (180 measurements)

Score distribution of Per-GPU GPU-to-host bandwidth (nvbandwidth)

Per-GPU host-to-GPU bandwidth (nvbandwidth)​

Average physical PCIe slot ingest speed (GB/s) per single GPU utilizing CE (Copy Engine).

Benchmark details

Benchmark ID
nvbandwidth:slot:host_to_gpu
Unit of measurement
GB/s
Server types evaluated
180
Total measurements
180
Range of measurements
6.1 – 57.9 GB/s
Better
↑ Higher is better

Score distribution (180 measurements)

Score distribution of Per-GPU host-to-GPU bandwidth (nvbandwidth)

Per-GPU host-device latency (nvbandwidth)​

Average round-trip transit time (ns) between CPU and individual GPUs utilizing SMSM: Streaming MultiprocessorThe basic processing unit of an NVIDIA GPU, which runs the threads of a kernel in groups, in parallel. Each SM has its own cores, registers, shared memory, and caches, and a GPU has from a few to more than a hundred SMs, so its compute throughput scales with their number.CUDA programming guideGlossary (Streaming Multiprocessor).

Benchmark details

Benchmark ID
nvbandwidth:slot:latency
Unit of measurement
ns
Server types evaluated
180
Total measurements
180
Range of measurements
558.8 – 8.6K ns
Better
↓ Lower is better

Score distribution (180 measurements)

Score distribution of Per-GPU host-device latency (nvbandwidth)

Per-GPU device memory copy bandwidth (nvbandwidth)​

Average on-GPU memory copy bandwidth (GB/s) per single GPU (device-to-device copy within the same GPU's memory) utilizing CE (Copy Engine).

Benchmark details

Benchmark ID
nvbandwidth:slot:local_copy
Unit of measurement
GB/s
Server types evaluated
82
Total measurements
82
Range of measurements
13.1 – 3.2K GB/s
Better
↑ Higher is better
warning

A copy both reads and writes every byte, so the scores are roughly 35-45% of the memory bandwidth listed on GPU datasheets (e.g. ~1,490 vs 3,350 GB/s on H100 SXM), and should not be compared to those figures directly. Time-sliced virtual GPU profiles (e.g. T4-4Q) can report proportionally lower values than the full GPU. Only recorded with nvbandwidthnvbandwidthAn open-source NVIDIA tool that measures memory copy bandwidth and latency between the CPU and NVIDIA GPUs, between GPUs, and within one GPU. It copies with the GPU's copy engines or with kernels on its streaming multiprocessors, and reports one matrix per test, with a row and a column for each CPU or GPU taking part. It prints plain text or JSON, and an optional multinode build measures GPUs across several servers.GitHubGlossary v0.7 and later.

Score distribution (82 measurements)

Score distribution of Per-GPU device memory copy bandwidth (nvbandwidth)