Skip to main content

Usage

Disk Safety​

This benchmark decides for each disk whether it may write to it. Disks it may write to (read_write) are filled and overwritten, and discarded first if they support it. Their previous data is lost. Run it only on ephemeral servers, never on production or similar systems.

benchmark.py checks each disk in the following order:

  1. A disk that is read-only, used as swap, or held by another device is read_only. Holders include LVMLVM: Logical Volume ManagerThe Linux tool that pools disks or partitions into volume groups and divides these into logical volumes, which can be resized while in use. It also supports snapshots, and striping or mirroring across disks.man pageGlossary (Logical Volume Manager) volumes and RAIDRAID: Redundant Array of Independent DisksA way to combine several disks into one logical device, for redundancy, speed or both. Common levels are RAID 0 (striping), RAID 1 (mirroring) and RAID 5 or 6 (striping with parity); on Linux, the md driver implements RAID in software.WikipediaGlossary (Redundant Array of Independent Disks) arrays built with md.
  2. A disk with no mounted filesystem is read_write.
  3. A disk with a protected mount is read_only. The protected mounts are /, /boot, /boot/efi, /usr, /var, /home, /root, /etc, /opt, /srv, /tmp, anything under /snap/, and any mount at or above /var/lib/docker.
  4. With SC_STORAGE_UMOUNT=0, any other mounted disk is read_only.
  5. Otherwise, this benchmark unmounts the disk's filesystems, deepest first, without forcing the unmount. If any unmount fails, for example because a filesystem is busy, it remounts what it already unmounted and marks the disk read_only. Otherwise, the disk is read_write.

This benchmark never remounts a filesystem after unmounting it, and the write tests overwrite its data.

Before every write test, this benchmark checks the disks again and skips the test if a disk is mounted, used as swap, or held again. The fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency.fio docsGlossary tool also runs with allow_mounted_write=0, and read tests run with fio --readonly.

The mount checks read the host's mount table from /proc/1/mountinfo and unmount through nsenter in the host's mount namespace. Both need --pid=host. This benchmark does not check if it runs in the host's PIDPID: Process IdentifierThe number by which the operating system identifies a running process. On Linux, a PID is assigned when a process starts and can be reused after it exits, and commands such as kill use it to target a process.WikipediaGlossary (Process Identifier) namespace.

Note that SC_STORAGE_UMOUNT=0 does not protect the disks that step 2 marks read_write: this benchmark always writes to them. There is no setting for a read-only run.

Production Setup​

Production runs the container with the following settings:

  • --privileged, --pid=host, and host networking
  • a timeout of 1 hour
  • after the benchmarks that have a set priority, because this benchmark may unmount and overwrite free disks
  • Resource Tracker settings, so the tracker that wraps this benchmark streams the run's resource usage to the Spare Cores Sentinel platform

A plain docker run with --privileged --pid=host runs the same tests, but without host networking and without the Resource Tracker settings.

Production runs this benchmark on ephemeral servers, on every server type with bundled storage: the inspector_rules.py file in sc-crawler skips only server types without it. The image does not check whether it runs on an ephemeral server, and does not tell bundled disks apart from other disks.

Run Duration​

A run took about 21 minutes on GCPGCP: Google Cloud PlatformThe cloud computing platform of Google. It offers virtual machines (Compute Engine), object storage, managed databases, Kubernetes and machine learning services, among others.Google CloudGlossary (Google Cloud Platform) c4a-standard-16-lssd, with four local NVMeNVMe: Non-Volatile Memory ExpressThe interface specification for accessing non-volatile storage, most often SSDs, over PCI Express. It is designed for the parallelism of flash memory, with many deep command queues, and NVMe over Fabrics extends it to network transports.NVM ExpressGlossary (Non-Volatile Memory Express) drives, and about 22 minutes on OVHcloudOVHcloudA French cloud provider, founded in 1999, with data centers in Europe, North America and Asia-Pacific. It offers dedicated (bare-metal) servers, public cloud instances based on OpenStack, and private cloud.OVHcloudGlossary i1-90, with two NVMe drives. See gcp/c4a-standard-16-lssd and ovh/i1-90 for the run outputs.

The following settings make up most of the time:

  • each read test: 10s measured plus a 2s ramp
  • each write or mixed test: 30s measured plus a 2s ramp
  • the sequential fill: up to 300s in total, split evenly between the groups that have writable disks
  • the random-write precondition: 120s per group with writable disks

Before running, this benchmark estimates the total test time, counting 2 seconds of fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency.fio docsGlossary overhead per test. If the estimate exceeds SC_STORAGE_BUDGET_S (2400s), it takes the following steps until the plan fits:

  1. Shorten read tests, down to SC_STORAGE_MIN_RUNTIME_S (5s).
  2. Shorten write tests, down to SC_STORAGE_MIN_WRITE_RUNTIME_S (15s).
  3. Drop the low-priority tests: randread_512, randwrite_512, randread_16m, randwrite_16m, randrw70_64k, and randrw70_1m.

Dropped tests are listed in skipped with the reason time budget. The estimate leaves out the discard and the health snapshots, so a run can take longer than the estimate. The config field records the runtimes actually used and the estimate.

Output​

This benchmark prints one JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary document to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary. It always exits with code 0 once the run completes; a failed test records its error in its own error field. Progress goes to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary, one timestamped line per step, with per-test IOPSIOPS: Input/Output Operations Per SecondThe number of read and write operations a storage device completes per second. It is the main measure of random-access performance, and depends on the block size, the queue depth and the mix of reads and writes.WikipediaGlossary (Input/Output Operations Per Second), MiB/s, and 99th-percentile latency in microseconds.

Top-Level Fields​

The JSON document has the following top-level fields:

  • schema_version (1), and the versions of fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency.fio docsGlossary, lsblk, smartctlsmartctlThe command-line tool of the smartmontools package that reads SMART data, self-test logs and device information from ATA, SCSI and NVMe disks, and can start self-tests.smartmontoolsGlossary (drive health queries), and nvme-cli (NVMe health logs)
  • started_at and finished_at, as UTCUTC: Coordinated Universal TimeThe primary time standard by which the world sets its clocks, the successor of Greenwich Mean Time. It has no time zone offset or daylight saving time, so UTC timestamps compare directly across regions.WikipediaGlossary (Coordinated Universal Time) timestamps
  • host: kernel, architecture, CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary count (nproc), and memory (mem_bytes)
  • config: the timing and budget settings used, including runtime_s and write_runtime_s after budget fitting, and estimated_s
  • inventory, excluded, and groups: the disks found and how they were grouped
  • benchmarks and preconditioning: the test records
  • health: the disk health snapshots
  • skipped: the tests that did not run

Disks and Groups​

Each inventory entry describes one disk, with lsblk and sysfs fields such as model, serial, size (bytes), tran (transport), rota (rotational), log_sec and phy_sec (sector sizes), write_cache, and disc_max (discard support). It also records how the disk was used and the decision made for it, in the following fields:

  • mountpoints, swap, and holders
  • mode (read_write or read_only) and mode_reason
  • unmounted: the filesystems this benchmark unmounted
  • group: the ID of the disk's group

The excluded field lists the devices that are not tested, each with a reason. The following devices are excluded:

  • devices that are not disks
  • virtual devices, such as loop, zramzramA Linux kernel module that creates compressed block devices in memory. It is most often used as swap, so swapped-out pages are compressed in RAM instead of written to disk, trading CPU time for memory.Linux kernel docsGlossary, md, or dm-
  • removable devices
  • empty devices
  • zoned devices

Disks with the same model, size, and subsystem form a group. Each groups entry has its key, members, representative (the first writable member by name, or the first member if none is writable), and mode.

Test Records​

Each benchmarks and preconditioning entry is one fio run, with the following fields:

  • test: <kind>_<block size>_qd<queue depth>. The kinds are randread, seqread, seqwrite, randwrite, randrw70 (70% reads), fill, and precondition_randwrite. The queue depth is the effective number of I/OI/O: Input/OutputThe transfer of data between a computer and the outside world, such as reading and writing disks or sending and receiving network packets. A workload limited by its speed rather than by the CPU is called I/O-bound.WikipediaGlossary (Input/Output) requests in flight per disk, numjobs times iodepth, which can be lower than the nominal queue depth. See fio Settings.
  • group, scope (single or group), and devices
  • rw, bs (bytes), rwmixread, numjobs, and iodepth
  • runtime_s, ramp_time_s, elapsed_s, and working_set_bytes
  • error: null, or the fio error
  • read and write: the results per direction, or null if the test did not use that direction
  • cpu: the summed user and system CPU percentages of the fio jobs
  • disk_util: the kernel's disk utilization statistics

Group tests also have per_device, with read and write results for each disk. Fill records also have written_bytes, purge (the discard result), and filled_fraction: the share of each disk the fill wrote, to four decimal places.

Each read and write result has the following fields:

  • iops, bw_bytes (bytes per second), io_bytes, total_ios, and runtime_ms
  • lat_ns: end-to-end latency per I/O in nanoseconds, with min, max, mean, stddev, samples, and percentiles at 1, 5, 10, 25, 50, 75, 90, 95, 99, 99.9, and 99.99, keyed as strings such as "99"
  • iops_interval and bw_bytes_interval: statistics over 500 ms windows, with min, p5, p50, p95, max, mean, tail_mean, and samples

tail_mean is the mean of the second half of the run. It leaves out the initial burst and includes the time a drive may still spend draining its write buffer.

Multi-job and multi-disk results are merged: IOPS and bandwidth are summed, and latency percentiles come from the merged fio latency histograms. Preconditioning records also have iops_series and bw_bytes_series, averaged over steps of 5 seconds (series_step_ms).

Health Snapshots​

The health field has one entry per disk, with the following fields:

  • before and after: snapshots of smartctl -x and, for NVMe drives, nvme smart-log, each with taken_at
  • nvme_id_ctrl and nvme_id_ns: the NVMe controller and namespace identification, before the run
  • delta: for NVMe drives, the change in counters over the run, such as bytes_read, bytes_written, media_errors, and controller_busy_time, plus percent_used_after, avail_spare_after, and critical_warning_after

Health data is best effort. Virtual disks often refuse these queries; a failed query is recorded as {"error": "..."} and does not fail the run.

Skipped Tests​

Each skipped entry has group, scope, test, and reason, with one of the following reasons:

  • bs < log_sec (<n>): the block size is below the disk's logical sector size
  • read_only: <mode_reason>: write tests on a disk this benchmark may not write to
  • fewer than 2 writable members: group write tests in a group with fewer than two writable disks
  • time budget: dropped to fit the time budget
  • fill failed: every later test on a disk whose fill failed
  • <disk> is in use again: a write test on a disk that was mounted, used as swap, or held again

Environment Variables​

VariableMeaningDefault
SC_STORAGE_RUNTIME_SMeasured seconds per read test.10
SC_STORAGE_WRITE_RUNTIME_SMeasured seconds per write or mixed test.30
SC_STORAGE_RAMP_SRamp seconds per test, not counted in the results; not used for the fill and the random-write precondition.2
SC_STORAGE_MIN_RUNTIME_SShortest read test when fitting the time budget.5
SC_STORAGE_MIN_WRITE_RUNTIME_SShortest write test when fitting the time budget.15
SC_STORAGE_FILL_BUDGET_STotal sequential-fill seconds, split evenly between fill steps.300
SC_STORAGE_WS_TARGET_BYTESBytes to fill per disk; 0 fills the whole disk.0
SC_STORAGE_RAND_PRECOND_SRandom-write precondition seconds.120
SC_STORAGE_BUDGET_STarget for the estimated total test time in seconds.2400
SC_STORAGE_UMOUNT0 keeps every mounted disk read-only instead of unmounting it.1
SC_STORAGE_PURGE0 skips the discard before the fill.1

An empty value uses the default. The SC_STORAGE_UMOUNT and SC_STORAGE_PURGE settings are off only when set to exactly 0.