Skip to main content

Storage

This benchmark measures the raw block device performance of a cloud server's bundled disks with fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency.fio docsGlossary. It finds the host's disks, groups identical ones, and runs random, sequential, and mixed read and write tests on one disk per group. Groups of two or more disks also run a few tests on all their disks at once. All I/OI/O: Input/OutputThe transfer of data between a computer and the outside world, such as reading and writing disks or sending and receiving network packets. A workload limited by its speed rather than by the CPU is called I/O-bound.WikipediaGlossary (Input/Output) goes directly to the raw device, bypassing the filesystem and the page cache.

This benchmark reports IOPSIOPS: Input/Output Operations Per SecondThe number of read and write operations a storage device completes per second. It is the main measure of random-access performance, and depends on the block size, the queue depth and the mix of reads and writes.WikipediaGlossary (Input/Output Operations Per Second), bandwidth, and latency percentiles for each test, at block sizes from 512 B to 16 MiB. It also records a disk inventory and before-and-after health snapshots. It overwrites every disk it is allowed to write to, so run it only on ephemeral servers.

Purpose​

Spare Cores Navigator publishes empirical performance measurements for over 5,000 cloud server types, such as raw CPU speed, memory bandwidth, and LLM inference speed. This benchmark adds storage performance: the Navigator will publish bandwidth and IOPS at different block sizes, both per disk and for each group of identical disks.

This benchmark focuses on a server type's bundled storage: the disk or disks provisioned with the server type by default, regardless of whether an HDDHDD: Hard Disk DriveA storage device that reads and writes data with magnetic heads on spinning platters. It offers cheap capacity, but each random access waits for a head to move and a platter to turn, so it completes far fewer random operations per second than an SSD.WikipediaGlossary (Hard Disk Drive), an SSDSSD: Solid-State DriveA storage device that keeps data in flash memory instead of on spinning disks, so it reads and writes much faster than an HDD, especially at random locations. A controller with its own firmware maps logical blocks to flash pages, and many SSDs write incoming data to a fast cache first, then move it to slower flash. Cloud servers use SSDs as local disks and behind network block storage.WikipediaGlossary, or network-attached storage backs them. Results reflect the storage as provisioned, including any provider-imposed IOPS or bandwidth limit.

Network storage attached on demand is out of scope, because its performance is not a property of the server type. See Disk Scope.

SSDs handle short bursts of writes faster than sustained writes, because they first write to fast caches and to flash that holds no data yet. To make results comparable across vendors, this benchmark preconditions each writable disk before measuring it, and runs write tests for longer than read tests. See Preconditioning for the steps.

Limitations​

These results describe raw block devices under synthetic fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency.fio docsGlossary load, not filesystem or application performance. The following are the main limitations:

  • Destructive: This benchmark overwrites every disk that is not in use, and unmounts and overwrites disks whose mounts are not protected and can be unmounted. It never remounts them.
  • Raw Devices Only: Tests bypass the filesystem and the page cache, so results exclude filesystem overhead and caching.
  • Bundled Storage Is Not Detected: The image tests every real block device it finds, including any volume attached on demand.
  • Partial Preconditioning: The sequential fill is capped at 300s in total, which filled 26-33% of each NVMeNVMe: Non-Volatile Memory ExpressThe interface specification for accessing non-volatile storage, most often SSDs, over PCI Express. It is designed for the parallelism of flash memory, with many deep command queues, and NVMe over Fabrics extends it to network transports.NVM ExpressGlossary (Non-Volatile Memory Express) drive on the first production servers.
  • Disks in Use Are Read-Only: The boot disk and other disks in use get read tests only, on whatever data they already hold.
  • No Single Score: The output has one record per test, and no headline metric is defined yet.

Usage​

Run the image via Docker on an ephemeral server that can be wiped.

danger

Every filesystem this benchmark unmounts is destroyed, together with the data on every disk that is not in use. Never run it on production or similar systems.

Set SC_STORAGE_UMOUNT=0 to leave mounted disks alone; disks that are not in use are still overwritten. See Disk Safety for which disks this benchmark writes to.

docker run --rm --privileged --pid=host ghcr.io/sparecores/benchmark-storage:main > storage.json

Both flags are required. --privileged gives the container raw access to the host's disks. --pid=host lets it read the host's mount table from /proc/1/mountinfo and unmount filesystems in the host's mount namespace, which the disk safety checks depend on. The image does not check if it runs in the host's PIDPID: Process IdentifierThe number by which the operating system identifies a running process. On Linux, a PID is assigned when a process starts and can be reused after it exits, and commands such as kill use it to target a process.WikipediaGlossary (Process Identifier) namespace.

See Production Setup for how production runs differ.

A run took about 21-22 minutes on the first production servers. This benchmark shortens or drops tests when its estimate exceeds 2400 seconds. See Run Duration for how the time is spent.

Settings​

The following are the common settings; see the full environment-variable reference for all options and defaults:

  • SC_STORAGE_UMOUNT=0 keeps every mounted disk read-only instead of unmounting it.
  • SC_STORAGE_PURGE=0 skips the discard (blkdiscardblkdiscardA Linux command from util-linux that discards (trims) a range of blocks, by default all blocks of a device, and so deletes their data. An SSD then treats those blocks as unwritten and can erase them in the background, which restores the write speed of a fresh drive.man pageGlossary) before the fill.
  • SC_STORAGE_RUNTIME_S and SC_STORAGE_WRITE_RUNTIME_S set the measured seconds per read test (default: 10) and per write test (default: 30).
  • SC_STORAGE_FILL_BUDGET_S sets the total sequential-fill time (default: 300), and SC_STORAGE_BUDGET_S the target for the estimated test time (default: 2400).

Results​

This benchmark prints one JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary document to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary when the run ends, and a timestamped progress log with per-test IOPS, MiB/s, and 99th-percentile latency to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary. The JSON document's main fields are the following:

  • inventory lists each disk with its model, size, and transport. The mode field is read_write or read_only, and mode_reason says why.
  • groups lists the groups of identical disks, each with its members and the representative that runs the single-disk tests.
  • Each benchmarks[] entry is one test, named <kind>_<block size>_qd<queue depth>, for example randread_4k_qd256. The scope field is single or group.
  • read and write hold iops, bw_bytes (bytes per second), and lat_ns, whose percentiles are latencies in nanoseconds. The iops_interval and bw_bytes_interval fields summarize 500 ms windows, including tail_mean, the mean of the second half of the run.
  • preconditioning holds the fill and random-write steps. The filled_fraction field shows how much of each disk the fill wrote.
  • skipped lists every test that did not run, with the reason.

See Output for every field and its unit.

Workloads​

benchmark.py runs the following kinds of tests for each group of identical disks:

  • Preconditioning: Discards and fills each writable disk sequentially, and later writes it randomly before the random-write tests.
  • Single-Disk Tests: Run random reads and writes at eight block sizes, sequential reads and writes, and 70/30 mixed I/O on one representative disk.
  • Group Tests: Run random and sequential reads on all disks of a group at once, and writes when at least two disks are writable.

For more information on test order and fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency.fio docsGlossary settings, see Workloads.

Design History​

The first version measured disks without preconditioning them. Partial fills and short write runs measured the drives' write-buffer burst rather than their sustained performance. The current version discards and fills each writable disk, runs write tests for longer than read tests (30 seconds compared with 10 seconds), and records best-effort health snapshots, so that results are comparable across vendors.

The fill is still capped by time. On the first production servers, it wrote about a quarter to a third of each NVMe drive. See Partial Fill.

See the changelog for the changes to this benchmark.

References​

See the full list of references used in this documentation.