Workloads
Disk Discovery and Grouping
benchmark.py lists the host's block devices with
lsblk, and skips the following devices:
- devices that are not disks
- virtual devices: names starting with
zramzramA Linux kernel module that creates compressed block devices in memory. It is most often used as swap, so swapped-out pages are compressed in RAM instead of written to disk, trading CPU time for memory.Linux kernel docsGlossary,ram,loop,nbd,md,dm-,sr, orfd, or devices with no real subsystem - removable devices
- empty devices
- zoned devices
Disks with the same model, size, and subsystem form a group. One representative per group, the first writable disk by name (or the first disk if none is writable), runs the full set of single-disk tests.
Groups of two or more disks also run group tests on all their disks at once,
which measure the combined throughput. Each disk is either written to
(read_write) or only read (read_only), as described in
Disk Safety.
The groups run one after the other, and so do the tests within a group. Each group runs its steps in the following order:
- Sequential fill of the writable disks (preconditioning)
- Single-disk read tests
- Group read tests
- Sequential write tests, single-disk and then group
- Random-write precondition of the writable disks
- Single-disk random-write and mixed tests
- Group random-write and mixed tests
Read tests run on the freshly filled disk. Random writes run last, because the random-write precondition fragments the SSDSSD: Solid-State DriveA storage device that keeps data in flash memory instead of on spinning disks, so it reads and writes much faster than an HDD, especially at random locations. A controller with its own firmware maps logical blocks to flash pages, and many SSDs write incoming data to a fast cache first, then move it to slower flash. Cloud servers use SSDs as local disks and behind network block storage.WikipediaGlossary's FTLFTL: Flash Translation LayerThe firmware layer of an SSD controller that maps the logical blocks the host sees to physical flash pages. Because flash has to be erased in large blocks before it is rewritten, the FTL writes new data to free pages, and runs garbage collection and wear leveling in the background. Wikipedia_and_mapping)Glossary (Flash Translation Layer) mapping, and a sequential test that ran after it would inherit that state.
A group whose disks are all read-only runs only steps 2 and 3, on whatever data the disks already hold.
Preconditioning
SSDsSSD: Solid-State DriveA storage device that keeps data in flash memory instead of on spinning disks, so it reads and writes much faster than an HDD, especially at random locations. A controller with its own firmware maps logical blocks to flash pages, and many SSDs write incoming data to a fast cache first, then move it to slower flash. Cloud servers use SSDs as local disks and behind network block storage.WikipediaGlossary first write incoming data to fast storage: a DRAMDRAM: Dynamic Random-Access MemoryThe most common type of volatile computer memory, which stores each bit as a charge in a capacitor that has to be refreshed periodically. Servers use it as main memory, and many storage devices as a cache.WikipediaGlossary (Dynamic Random-Access MemoryRAM: Random-Access MemoryThe main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off.WikipediaGlossary) or SLCSLC: Single-Level CellFlash memory that stores one bit per cell, which makes it faster and more durable than flash storing more bits per cell, such as TLC or QLC. Many SSDs run part of their flash in SLC mode as a write cache, and write more slowly once a long write fills it.WikipediaGlossary (Single-Level Cell) cache, and spare area, which is flash that holds no data yet. On a drive that is only partly written, new writes keep finding spare area, so the drive never has to run garbage collectiongarbage collectionThe background work in which an SSD frees flash for new writes. Flash can only be erased in large blocks, so the drive copies the still-valid pages out of a block, then erases the whole block. It runs when spare area runs low, and it slows the writes that arrive meanwhile, which is why an SSD's sustained write speed is lower than its burst speed. Programming languages use the same term for reclaiming unused memory.WikipediaGlossary, and short tests report burst speeds rather than sustained ones. This benchmark prepares each writable disk in the following steps:
- Discard:
blkdiscardblkdiscardA Linux command from util-linux that discards (trims) a range of blocks, by default all blocks of a device, and so deletes their data. An SSD then treats those blocks as unwritten and can erase them in the background, which restores the write speed of a fresh drive.man pageGlossary trims the whole disk, so every run starts from the same mapping state. Disks without discard support skip this step, andSC_STORAGE_PURGE=0turns it off. - Sequential fill: one
fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, andfioreports the resulting IOPS, bandwidth, and latency.fio docsGlossary job per disk writes it sequentially in 1 MiB blocks at a queue depth of 64, until the disk is full, the fill reachesSC_STORAGE_WS_TARGET_BYTES, or the fill time runs out. The fill time isSC_STORAGE_FILL_BUDGET_S(300s), split evenly between the groups with writable disks. - Random-write precondition: after the read and sequential write tests,
4 KiB random writes at a queue depth of 256 run for
SC_STORAGE_RAND_PRECOND_S(120s).
Both preconditioning steps are reported under preconditioning, not
benchmarks, with IOPSIOPS: Input/Output Operations Per SecondThe number of read and write operations a storage device completes per second. It is the main measure of random-access performance, and depends on the block size, the queue depth and the mix of reads and writes.WikipediaGlossary (Input/Output Operations Per Second) and bandwidth time
series in steps of 5 seconds.
After the fill, every later test on the disk uses only the region the fill wrote. If the fill fails or writes nothing, every later test on that disk is skipped, because reads would hit unmapped blocks. Read-only disks always use the whole disk.
Write and mixed tests run for 30 seconds, compared with 10 seconds for read tests, to get past the drive's DRAM or SLC write buffer.
Single-Disk Tests
The representative of each group runs the following read tests:
- random reads of 4 KiB at a queue depth of 1
- random reads at a queue depth of 256, at each block size: 512 B, 4 KiB, 16 KiB, 64 KiB, 256 KiB, 1 MiB, 4 MiB, and 16 MiB
- sequential reads of 1 MiB at a queue depth of 64
If the representative is writable, it also runs the following write tests:
- sequential writes of 1 MiB at a queue depth of 64
- mixed random I/OI/O: Input/OutputThe transfer of data between a computer and the outside world, such as reading and writing disks or sending and receiving network packets. A workload limited by its speed rather than by the CPU is called I/O-bound.WikipediaGlossary (Input/Output) with 70% reads at a queue depth of 256, at 4 KiB, 64 KiB, and 1 MiB
- mixed random I/O with 70% reads, at 4 KiB and a queue depth of 1
- random writes of 4 KiB at a queue depth of 1
- random writes at a queue depth of 256, at each block size from 512 B to 16 MiB
Block sizes below the disk's logical sector size are skipped. For example, a disk with 4 KiB sectors skips the 512 B tests.
Group Tests
A group of two or more disks runs the following read tests on all its disks at once, including read-only ones:
- random reads of 4 KiB at a queue depth of 256
- sequential reads of 1 MiB at a queue depth of 64
If at least two disks of the group are writable, those disks also run the following write tests:
- sequential writes of 1 MiB at a queue depth of 64
- mixed random I/O with 70% reads, at 4 KiB and a queue depth of 256
- random writes of 4 KiB at a queue depth of 256
Group results add up all disks, and per_device holds each disk's share.
fio Settings
All tests use the libaio I/O engine with direct=1, so I/O bypasses the page
cache. Each test runs for a fixed time (time_based=1) after a ramp of
2 seconds. The sequential fill instead runs until it has written its target
(the whole disk by default) or its time runs out, and neither preconditioning
step has a ramp.
The queue depths above are the nominal number of I/Os in flight per disk.
benchmark.py splits them across jobs as follows:
- Tests with a queue depth of 1 and the fill use one job per disk. Other tests use up to four jobs per disk, and no more than the CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary count divided by the number of disks in the test.
- Each job's
iodepthis the queue depth divided by the number of jobs. iodepthis then capped so that no test has more than 256 MiB in flight across all its jobs and disks.
Because of this cap, large block sizes run at a lower queue depth. The test
name shows the effective queue depth. For example, on a server with 16 vCPUsvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary,
4 MiB random reads run as randread_4m_qd64 and 16 MiB random reads
as randread_16m_qd16.
Sequential tests with more than one job split the disk into equal stripes, one
per job. Random tests use the tausworthe64 generator without a random map
(norandommap=1) and without a repeatable seed (randrepeat=0).
The fiofioThe Flexible I/O Tester, an open-source storage benchmark and I/O workload generator. Each job describes an access pattern, such as random or sequential reads and writes, the block size, the queue depth, and the I/O engine, and fio reports the resulting IOPS, bandwidth, and latency.fio docsGlossary tool measures the end-to-end latency of each I/O, and this benchmark
reports its percentiles from fio's latency histograms. It also logs IOPS and
bandwidth every 500 ms, which give the interval statistics in the output.
See Output for the fields.