Usage
Disk Safety
This benchmark decides for each disk whether it may write to it. Disks it may
write to (read_write) are filled and overwritten, and discarded first if they
support it. Their previous data is lost. Run it only on ephemeral servers,
never on production or similar systems.
benchmark.py checks each disk in the following order:
- A disk that is read-only, used as swap, or held by another device is
read_only. Holders include LVMLVM: Logical Volume ManagerThe Linux tool that pools disks or partitions into volume groups and divides these into logical volumes, which can be resized while in use. It also supports snapshots, and striping or mirroring across disks.man pageGlossary (Logical Volume Manager) volumes and RAIDRAID: Redundant Array of Independent DisksA way to combine several disks into one logical device, for redundancy, speed or both. Common levels are RAID 0 (striping), RAID 1 (mirroring) and RAID 5 or 6 (striping with parity); on Linux, themddriver implements RAID in software.WikipediaGlossary (Redundant Array of Independent Disks) arrays built withmd. - A disk with no mounted filesystem is
read_write. - A disk with a protected mount is
read_only. The protected mounts are/,/boot,/boot/efi,/usr,/var,/home,/root,/etc,/opt,/srv,/tmp, anything under/snap/, and any mount at or above/var/lib/docker. - With
SC_STORAGE_UMOUNT=0, any other mounted disk isread_only. - Otherwise, this benchmark unmounts the disk's filesystems, deepest first,
without forcing the unmount. If any unmount fails, for example because a
filesystem is busy, it remounts what it already unmounted and marks the
disk
read_only. Otherwise, the disk isread_write.
This benchmark never remounts a filesystem after unmounting it, and the write tests overwrite its data.
Before every write test, this benchmark checks the disks again and skips the
test if a disk is mounted, used as swap, or held again. The fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency.fio docsGlossary tool also runs with
allow_mounted_write=0, and read tests run with fio --readonly.
The mount checks read the host's mount table from /proc/1/mountinfo and
unmount through nsenter in the host's mount namespace. Both need
--pid=host. This benchmark does not check if it runs in the host's PIDPID: Process IdentifierThe number by which the operating system identifies a running process. On Linux, a PID is assigned when a process starts and can be reused after it exits, and commands such as kill use it to target a process.WikipediaGlossary
(Process Identifier) namespace.
Note that SC_STORAGE_UMOUNT=0 does not protect the disks that step 2 marks
read_write: this benchmark always writes to them. There is no setting for a
read-only run.
Production Setup
Production runs the container with the following settings:
--privileged,--pid=host, and host networking- a timeout of 1 hour
- after the benchmarks that have a set priority, because this benchmark may unmount and overwrite free disks
- Resource Tracker settings, so the tracker that wraps this benchmark streams the run's resource usage to the Spare Cores Sentinel platform
A plain docker run with --privileged --pid=host runs the same tests, but
without host networking and without the Resource Tracker settings.
Production runs this benchmark on ephemeral servers, on every server type with
bundled storage: the
inspector_rules.py
file in sc-crawler skips only server types without it. The image does not
check whether it runs on an ephemeral server, and does not tell bundled disks
apart from other disks.
Run Duration
A run took about 21 minutes on GCPGCP: Google Cloud PlatformThe cloud computing platform of Google. It offers virtual machines (Compute Engine), object storage, managed databases, Kubernetes and machine learning services, among others.Google CloudGlossary (Google Cloud Platform)
c4a-standard-16-lssd, with four local NVMeNVMe: Non-Volatile Memory ExpressThe interface specification for accessing non-volatile storage, most often SSDs, over PCI Express. It is designed for the parallelism of flash memory, with many deep command queues, and NVMe over Fabrics extends it to network transports.NVM ExpressGlossary (Non-Volatile Memory Express)
drives, and about 22 minutes on OVHcloudOVHcloudA French cloud provider, founded in 1999, with data centers in Europe, North America and Asia-Pacific. It offers dedicated (bare-metal) servers, public cloud instances based on OpenStack, and private cloud.OVHcloudGlossary i1-90, with two NVMe drives. See
gcp/c4a-standard-16-lssd
and
ovh/i1-90
for the run outputs.
The following settings make up most of the time:
- each read test: 10s measured plus a 2s ramp
- each write or mixed test: 30s measured plus a 2s ramp
- the sequential fill: up to 300s in total, split evenly between the groups that have writable disks
- the random-write precondition: 120s per group with writable disks
Before running, this benchmark estimates the total test time, counting 2 seconds
of fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency.fio docsGlossary overhead per test. If the estimate exceeds SC_STORAGE_BUDGET_S
(2400s), it takes the following steps until the plan fits:
- Shorten read tests, down to
SC_STORAGE_MIN_RUNTIME_S(5s). - Shorten write tests, down to
SC_STORAGE_MIN_WRITE_RUNTIME_S(15s). - Drop the low-priority tests:
randread_512,randwrite_512,randread_16m,randwrite_16m,randrw70_64k, andrandrw70_1m.
Dropped tests are listed in skipped with the reason time budget. The
estimate leaves out the discard and the health snapshots, so a run can take
longer than the estimate. The config field records the runtimes actually used
and the estimate.
Output
This benchmark prints one JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary document to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary. It always exits with code 0
once the run completes; a failed test records its error in its own error
field. Progress goes to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary, one timestamped line per step, with per-test
IOPSIOPS: Input/Output Operations Per SecondThe number of read and write operations a storage device completes per second. It is the main measure of random-access performance, and depends on the block size, the queue depth and the mix of reads and writes.WikipediaGlossary (Input/Output Operations Per Second), MiB/s, and 99th-percentile latency
in microseconds.
Top-Level Fields
The JSON document has the following top-level fields:
schema_version(1), and the versions offiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, andfioreports the resulting IOPS, bandwidth, and latency.fio docsGlossary,lsblk,smartctlsmartctlThe command-line tool of the smartmontools package that reads SMART data, self-test logs and device information from ATA, SCSI and NVMe disks, and can start self-tests.smartmontoolsGlossary (drive health queries), andnvme-cli(NVMe health logs)started_atandfinished_at, as UTCUTC: Coordinated Universal TimeThe primary time standard by which the world sets its clocks, the successor of Greenwich Mean Time. It has no time zone offset or daylight saving time, so UTC timestamps compare directly across regions.WikipediaGlossary (Coordinated Universal Time) timestampshost: kernel, architecture, CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary count (nproc), and memory (mem_bytes)config: the timing and budget settings used, includingruntime_sandwrite_runtime_safter budget fitting, andestimated_sinventory,excluded, andgroups: the disks found and how they were groupedbenchmarksandpreconditioning: the test recordshealth: the disk health snapshotsskipped: the tests that did not run
Disks and Groups
Each inventory entry describes one disk, with lsblk and sysfs fields such
as model, serial, size (bytes), tran (transport), rota (rotational),
log_sec and phy_sec (sector sizes), write_cache, and disc_max (discard
support). It also records how the disk was used and the decision made for it,
in the following fields:
mountpoints,swap, andholdersmode(read_writeorread_only) andmode_reasonunmounted: the filesystems this benchmark unmountedgroup: the ID of the disk's group
The excluded field lists the devices that are not tested, each with a reason.
The following devices are excluded:
- devices that are not disks
- virtual devices, such as
loop,zramzramA Linux kernel module that creates compressed block devices in memory. It is most often used as swap, so swapped-out pages are compressed in RAM instead of written to disk, trading CPU time for memory.Linux kernel docsGlossary,md, ordm- - removable devices
- empty devices
- zoned devices
Disks with the same model, size, and subsystem form a group. Each groups
entry has its key, members, representative (the first writable member by
name, or the first member if none is writable), and mode.
Test Records
Each benchmarks and preconditioning entry is one fio run, with the
following fields:
test:<kind>_<block size>_qd<queue depth>. The kinds arerandread,seqread,seqwrite,randwrite,randrw70(70% reads),fill, andprecondition_randwrite. The queue depth is the effective number of I/OI/O: Input/OutputThe transfer of data between a computer and the outside world, such as reading and writing disks or sending and receiving network packets. A workload limited by its speed rather than by the CPU is called I/O-bound.WikipediaGlossary (Input/Output) requests in flight per disk,numjobstimesiodepth, which can be lower than the nominal queue depth. SeefioSettings.group,scope(singleorgroup), anddevicesrw,bs(bytes),rwmixread,numjobs, andiodepthruntime_s,ramp_time_s,elapsed_s, andworking_set_byteserror:null, or thefioerrorreadandwrite: the results per direction, ornullif the test did not use that directioncpu: the summed user and system CPU percentages of thefiojobsdisk_util: the kernel's disk utilization statistics
Group tests also have per_device, with read and write results for each
disk. Fill records also have written_bytes, purge (the discard result), and
filled_fraction: the share of each disk the fill wrote, to four decimal
places.
Each read and write result has the following fields:
iops,bw_bytes(bytes per second),io_bytes,total_ios, andruntime_mslat_ns: end-to-end latency per I/O in nanoseconds, withmin,max,mean,stddev,samples, andpercentilesat 1, 5, 10, 25, 50, 75, 90, 95, 99, 99.9, and 99.99, keyed as strings such as"99"iops_intervalandbw_bytes_interval: statistics over 500 ms windows, withmin,p5,p50,p95,max,mean,tail_mean, andsamples
tail_mean is the mean of the second half of the run. It leaves out the
initial burst and includes the time a drive may still spend draining its write
buffer.
Multi-job and multi-disk results are merged: IOPS and bandwidth are summed, and
latency percentiles come from the merged fio latency histograms.
Preconditioning records also have iops_series and bw_bytes_series, averaged
over steps of 5 seconds (series_step_ms).
Health Snapshots
The health field has one entry per disk, with the following fields:
beforeandafter: snapshots ofsmartctl -xand, for NVMe drives,nvme smart-log, each withtaken_atnvme_id_ctrlandnvme_id_ns: the NVMe controller and namespace identification, before the rundelta: for NVMe drives, the change in counters over the run, such asbytes_read,bytes_written,media_errors, andcontroller_busy_time, pluspercent_used_after,avail_spare_after, andcritical_warning_after
Health data is best effort. Virtual disks often refuse these queries; a failed
query is recorded as {"error": "..."} and does not fail the run.
Skipped Tests
Each skipped entry has group, scope, test, and reason, with one of
the following reasons:
bs < log_sec (<n>): the block size is below the disk's logical sector sizeread_only: <mode_reason>: write tests on a disk this benchmark may not write tofewer than 2 writable members: group write tests in a group with fewer than two writable diskstime budget: dropped to fit the time budgetfill failed: every later test on a disk whose fill failed<disk> is in use again: a write test on a disk that was mounted, used as swap, or held again
Environment Variables
| Variable | Meaning | Default |
|---|---|---|
SC_STORAGE_RUNTIME_S | Measured seconds per read test. | 10 |
SC_STORAGE_WRITE_RUNTIME_S | Measured seconds per write or mixed test. | 30 |
SC_STORAGE_RAMP_S | Ramp seconds per test, not counted in the results; not used for the fill and the random-write precondition. | 2 |
SC_STORAGE_MIN_RUNTIME_S | Shortest read test when fitting the time budget. | 5 |
SC_STORAGE_MIN_WRITE_RUNTIME_S | Shortest write test when fitting the time budget. | 15 |
SC_STORAGE_FILL_BUDGET_S | Total sequential-fill seconds, split evenly between fill steps. | 300 |
SC_STORAGE_WS_TARGET_BYTES | Bytes to fill per disk; 0 fills the whole disk. | 0 |
SC_STORAGE_RAND_PRECOND_S | Random-write precondition seconds. | 120 |
SC_STORAGE_BUDGET_S | Target for the estimated total test time in seconds. | 2400 |
SC_STORAGE_UMOUNT | 0 keeps every mounted disk read-only instead of unmounting it. | 1 |
SC_STORAGE_PURGE | 0 skips the discard before the fill. | 1 |
An empty value uses the default. The SC_STORAGE_UMOUNT and SC_STORAGE_PURGE
settings are off only when set to exactly 0.