Limitations
Disclaimer: What This Benchmark Does Not Deliver
A Production Workload
Transactions are synthetic proxies: deliberately balanced mixes of PostgreSQLPostgreSQLAn open-source object-relational database system, developed by a global community and descended from the POSTGRES project at UC Berkeley in 1986. It uses and extends SQL, with transactions, many index types, extensions, and JSON data.PostgreSQL docsGlossary subsystems. See Workloads for details.
This benchmark does not predict the specific performance of any given application. It instead gives a general sense of the relative RDBMSRDBMS: Relational Database Management SystemA database system that stores data in tables of rows and columns, related through keys and queried with SQL. It keeps data consistent with constraints and transactions; PostgreSQL, MySQL and Oracle Database are examples.WikipediaGlossary (Relational Database Management System) performance expected of a server type.
Disk I/O Speed
The scores do not measure storage I/OI/O: Input/OutputThe transfer of data between a computer and the outside world, such as reading and writing disks or sending and receiving network packets. A workload limited by its speed rather than by the CPU is called I/O-bound.WikipediaGlossary (Input/Output) performance. Database throughput usually hinges first on disk IOPSIOPS: Input/Output Operations Per SecondThe number of read and write operations a storage device completes per second. It is the main measure of random-access performance, and depends on the block size, the queue depth and the mix of reads and writes.WikipediaGlossary (Input/Output Operations Per Second), then on bandwidth. In the cloud, that disk is almost always network-attached block storage, provisioned independently of the server type and entirely up to the user, so it says little about the server itself.
Deploying volumes with high-enough IOPS to never bottleneck across the more than 5,000 server types in the Spare Cores Navigator Web UI would also be prohibitively expensive. Because of this, we exclude disk speed from the measurement and score the database engine's CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary and memory performance.
Network Performance
The scores say nothing about a server's network throughput or latency. With a remote client, both bandwidth and especially RTTRTT: Round-Trip TimeThe time a network packet takes to reach its destination plus the time its response takes to get back. It sets a lower bound on the latency of every request-response exchange, and grows with the distance between the endpoints.WikipediaGlossary (Round-Trip Time) between client and server can dominate chatty workloads. Placing the client close to the server is not enough on its own: occasional latency glitches still distort lightweight-workload results.
The workload is therefore designed so the remaining RTT is a rounding error; see CPU-Heavy Transactions.
Uniform Data
Data distribution is uniform rather than ZipfianZipfianA skewed probability distribution, named after Zipf's law, in which an item's frequency is inversely proportional to its rank, so a few items are accessed far more often than the rest. Benchmarks use it to model real-world hot spots, such as popular products or users.WikipediaGlossary. The product catalog has
20,000 products, of which order items reference only the first 5,000, but
customer, order, and order-item values use g % k modular arithmetic rather
than a realistic power-law distribution with a few high-activity customers. See
Workloads for details.
Scope Decisions
Most published database benchmarks focus on storage, network throughput, a single database operation, or one production workload. Our approach focuses on the CPU and memory speed of the server instead: we keep the workload and the PostgreSQL major version constant and vary the hardware across thousands of server types.
Minor Engine Versions
Only the PostgreSQL major version is fixed. DBaaSDBaaS: Database as a ServiceA cloud service model in which the vendor runs a database for its users, handling provisioning, backups, patching and replication. Examples are Amazon RDS and Google Cloud SQL.WikipediaGlossary (Database as a Service)
providers may apply minor PostgreSQL upgrades automatically. This benchmark does
not check or enforce the version of a remote target; version selection and
pinning are deployment responsibilities. Standalone runs use whichever
PostgreSQL 18 minor release the postgres:18 base image had when the image was
built.
Instance Size Range
From small instances (e.g. 1 vCPUvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary and 2 GiB of RAMRAM: Random-Access MemoryThe main memory of a computer, which holds the running programs and their data. It is much faster than disk storage, and its content is lost when the power is off.WikipediaGlossary) to large nodes with hundreds of vCPUs, the same workload must produce meaningful, comparable numbers.
HammerDBHammerDBAn open-source database benchmark engine. It runs standardized workloads, such as TPROC-C, with the same test definitions on SQL Server, Db2, Oracle, PostgreSQL, MariaDB, and MySQL, so that results are comparable across database systems. Runs can be scripted and repeated automatically.HammerDBGlossary TPROC-CTPROC-C: HammerDB's transaction processing workloadThe OLTP workload of the HammerDB benchmark tool, derived from the TPC-C specification of an order-entry system with warehouses, orders and payments. It is not an official TPC-C implementation, so its results are not comparable with published TPC-C results.HammerDB docsGlossary (HammerDB's transaction processing
workload) and similar suites size their
datasets by warehouse count, and pgbenchpgbenchThe benchmark tool shipped with PostgreSQL. It runs a SQL script repeatedly over many concurrent client sessions, and reports the throughput in transactions per second and the latency. Its built-in script is modeled on TPC-B, and custom scripts can define any other workload.PostgreSQL docsGlossary by scale factor. Neither sizing
scheme can fulfill this purpose.
Because of this, we chose a fixed-size workload that can run on every server type in the Navigator's Web UI with at least 2 GiB of RAM, with larger servers being taxed by concurrency rather than workload size.
Engine Configuration
The vendor tunes the managed engine, and the harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary cannot assume superuser
access or GUCGUC: Grand Unified ConfigurationThe configuration system of PostgreSQL, and by extension its individual settings, such as shared_buffers or work_mem. A setting can be set in postgresql.conf, on the command line, per database or role, or per session with SET.PostgreSQL docsGlossary (Grand Unified
Configuration)
control on DBaaS. For this benchmark, we deliberately do not tune the DBaaS
engine: the tuning of the managed service is part of what is being measured, so
the score reflects the vendor's configuration.
JIT and Parallel Query
For pgbench_ro, the harness disables PostgreSQL's JITJIT: Just-in-Time CompilationThe compilation of a program's code to machine code while the program runs, instead of before. The compiler can use what is known only at run time, such as the actual data, but the compile time adds to the run time, so it pays off mostly on long-running code. PostgreSQL, for example, compiles parts of long, CPU-bound analytical queries, such as their expressions, with LLVM.WikipediaPostgreSQL docsGlossary (Just-in-Time
Compilation) and parallel
query, and sets
work_mem to 64MB; see Workload Settings.
These settings are not applied to pgbench_tpcb.
This benchmark measures raw engine and CPU behavior, not JIT compilation variance or Gather scalability. Those are treated as a separate testing axis.
A Single Monolithic Statement
One SELECT whose eight query blocks are CTEs (Common Table
Expressions)
combined with UNION ALL is deliberate: it keeps one pgbench transaction
equal to one network round trip. This makes the pgbench_ro workload resilient
to RTT simulated with netemnetem: Network EmulatorA Linux traffic control queueing discipline that adds delay, jitter, packet loss, duplication or reordering to the packets leaving a network interface. It is used to test how software behaves on slow or unreliable networks.SRT cookbookGlossary (Network Emulator); see
Latency and pipelining for the
experiments.
The tradeoff is that per-block planner GUCs (e.g. forcing Merge Join specifically) are not possible without affecting every block.
Pre-Calibrated Weights
The time shares of the eight query blocks in the pgbench_ro transaction were
manually calibrated and remain fixed for all runs. See the
block timings
and the recalibration procedure for
details.
Design Constraints
With the help of the benchANT team, over many iterations with different tools and configurations, we identified the following core principles.
Memory-Fit, Small Dataset
This benchmark is designed to use a small dataset of about 303 MiB. On the smallest nodes, the dataset does not fit in memory, which can add some disk overhead, so production runs this benchmark only on servers with at least 2 GiB of RAM, as reported by the vendor.
There, the dataset is meant to stay in memory (shared_buffers plus the OSOS: operating systemThe software that manages a computer's hardware and runs programs on it, such as Linux or Windows. A cloud server runs the OS of the image it was started from.Glossary
page cache), so that the disk is not read again after warmup. Because the
dataset is small by design, large instances are exercised through concurrency
rather than data volume.
Read-Only Workload
The measured pgbench_ro transaction writes no data, so in the steady state of
this benchmark it generates no WALWAL: Write-Ahead LoggingA technique in which a database writes each change to a durable log before it changes the data files. After a crash, replaying the log restores a consistent state, and shipping it to other servers drives replication and point-in-time recovery. PostgreSQL, for example, calls its log the WAL, and on write-heavy workloads the speed of WAL writes often limits throughput.WikipediaPostgreSQL docsGlossary (Write-Ahead
Logging) records.
CPU-Heavy Transactions
Each transaction takes about 70 to 100 ms of server work per connection. With 1 to 5 ms of RTT against roughly 100 ms of server work, network time stays a small fraction of each transaction and cannot dominate. Experiments showed that lightweight read-only transactions are sensitive to network delay, so the default workload uses a heavier cached transaction; see Latency and pipelining for the results.