Redis
This benchmark measures Redis 7.4 SET throughput and latency with
memtier_benchmarkmemtier_benchmarkAn open-source load generator for Redis and Memcached, developed by Redis Ltd. It runs many concurrent connections over one or more threads, sends a configurable mix of SET and GET requests, and reports operations per second, latency averages, and latency percentiles per request type.memtier_benchmark on GitHubGlossary. It starts one redis-server and one memtier_benchmark
client for each CPUCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary, all in the same container and connected over loopback,
and runs every configuration for 10 seconds.
It reports the total operations per second and the average latency of the Redis SET commandRedis SET commandThe Redis command that stores a string value under a key, replacing any existing value. Options can set an expiry time, or store the value only if the key does or does not exist yet.Redis docsGlossary at six pipeline depths, from 1 to 512, together with the CPU time the Redis servers and the clients used. Spare Cores Navigator publishes these as Redis server+client speed, Redis server (extrapolated) speed, and Redis latency.
Purpose
This benchmark shows how fast a server type serves a simple in-memory key-value workload, and how throughput and latency change as each client connection sends more requests at once.
Because each RedisRedisAn open-source, in-memory key-value data store, used as a database, cache, and message broker. It keeps its data in RAM, supports data structures such as strings, hashes, lists, and sets, and can persist data to disk with periodic snapshots or an append-only log.Redis docsGlossary server shares its CPU with its memtier_benchmarkmemtier_benchmarkAn open-source load generator for Redis and Memcached, developed by Redis Ltd. It runs many concurrent connections over one or more threads, sends a configurable mix of SET and GET requests, and reports operations per second, latency averages, and latency percentiles per request type.memtier_benchmark on GitHubGlossary client,
this benchmark records their CPU time separately, so the servers' share of the
work can be separated from the clients' share. See Results for how
the Navigator uses it.
Limitations
These results compare RedisRedisAn open-source, in-memory key-value data store, used as a database, cache, and message broker. It keeps its data in RAM, supports data structures such as strings, hashes, lists, and sets, and can persist data to disk with periodic snapshots or an append-only log.Redis docsGlossary SET performance on one host, not the throughput of a production Redis deployment. The main limitations are the following:
- Shared CPUs: Each Redis server shares its CPU with a client, so
rpsis not the maximum a dedicated Redis server would reach. See Results for the CPU-time-adjusted score the Navigator publishes. - Workload: Only the SET command is measured, with
memtier_benchmarkmemtier_benchmarkAn open-source load generator for Redis and Memcached, developed by Redis Ltd. It runs many concurrent connections over one or more threads, sends a configurable mix of SET and GET requests, and reports operations per second, latency averages, and latency percentiles per request type.memtier_benchmark on GitHubGlossary's default key pattern, key range, value size, and connection count. GET and mixed workloads are not measured. - Loopback Only: Clients and servers run on the same host and talk over loopback, so this benchmark does not measure network throughput or latency.
- Untuned Servers: Each
redis-serverstarts with only a data directory and a port, so Redis 7.4.0 defaults apply, including persistence, memory, and threading settings. - Single Short Run: Each configuration runs once for 10s, with no warmup and no repetitions.
- CPU Count: The number of server and client pairs follows the CPUs the
container may run on. Docker's
--cpuset-cpuschanges it, but--cpusdoes not: with--cpus, this benchmark still starts one pair per host CPU. - Error Checks: The harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary does not check
memtier_benchmarkexit codes, and it continues even if some servers do not answer the flush before a configuration. A client that prints no result adds nothing torpsbut still counts in thelatencyaverage.
Usage
Run the image via Docker:
docker run --rm --privileged ghcr.io/sparecores/benchmark-redis:main
The harness runs the RedisRedisAn open-source, in-memory key-value data store, used as a database, cache, and message broker. It keeps its data in RAM, supports data structures such as strings, hashes, lists, and sets, and can persist data to disk with periodic snapshots or an append-only log.Redis docsGlossary servers and clients at the highest scheduling
priority (nice -n -20niceA Unix command that runs a program with a changed scheduling priority, from -20 (highest) to 19 (lowest). Raising a process's priority above the default needs the CAP_SYS_NICE capability, which root has.man pageGlossary), which needs the CAP_SYS_NICE capability that
--privileged grants. Production runs the container the same way and also
sets host networking (--network host); this benchmark's traffic itself stays
on loopback inside the container.
The container entrypointcontainer entrypointThe executable that a container image runs when a container starts, set by the ENTRYPOINT instruction of its Dockerfile. The image's default command (CMD), or the arguments given after the image name in docker run, are passed to it as arguments, so an entrypoint can wrap any command, for example to set up the environment or record resource usage first. The docker run --entrypoint option replaces it.Dockerfile referenceGlossary runs this benchmark under resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary, which
samples resource usage during the run. Its data is not published.
Measurement takes 60 seconds (six configurations of 10 seconds each), plus server startup and a flush of all servers before each configuration.
Settings
This benchmark reads no environment variables or command-line options. The
operation, pipeline depths, run time, and ports are fixed in
benchmark.py; see Workload. To change the
number of server and client pairs, limit the container's CPUs with
--cpuset-cpus.
Results
This benchmark prints a comma-separated table to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary: a header row with
quoted column names, then one row per pipeline depth. Nothing else is written
besides Redis data under /data in the container.
"operation","threads","pipeline","rps","latency","server_usr","server_sys","client_usr","client_sys"
The columns mean the following:
operationis always"SET", andthreadsis always1, thememtier_benchmarkmemtier_benchmarkAn open-source load generator for Redis and Memcached, developed by Redis Ltd. It runs many concurrent connections over one or more threads, sends a configurable mix of SET and GET requests, and reports operations per second, latency averages, and latency percentiles per request type.memtier_benchmark on GitHubGlossary thread count per client.pipelineis thememtier_benchmark--pipelinevalue: how many requests each connection sends before waiting for replies.rpsis the sum of SET operations per second over all clients, as an integer.latencyis the average SET latency in milliseconds: the mean of each client's average latency, not weighted by its operation count.server_usrandserver_sysare the user and system CPU seconds that all Redis servers used during the configuration.client_usrandclient_sysare the user and system CPU seconds of thememtier_benchmarkclients of the configuration.
For example, on an aws/m5.large (2 vCPUsvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary), rps rises from 69,962 at
pipeline 1 to 685,686 at pipeline 512, while latency rises from 1.43 ms to
74.36 ms
(production output).
The Navigator publishes the following three scores for each pipeline depth:
- Redis server+client speed:
rpsin operations per second. - Redis server (extrapolated) speed:
rpsmultiplied by the ratio of all CPU time (servers and clients) to the servers' CPU time in operations per second. - Redis latency:
latencyin milliseconds; lower is better.
Each score carries a note with the server and client CPU seconds. Production
records the Redis version from redis-server -v. The memtier_benchmark tool
is installed from the Redis package repository without a version pin, so a
rebuild of the image can change it, and production does not record its
version.
Workload
This benchmark runs one workload: a sweep of Redis pipeliningRedis pipeliningA technique in which a client sends several Redis commands without waiting for the reply to each one, then reads all replies together. It reduces the number of network round trips and system calls per command, which raises throughput, while each command waits longer for its reply.Redis docsGlossary depths for SET
operations. benchmark.py starts one redis-server per
CPU, on ports 7000 and up, each with its own data directory under /data. The
servers run until the container exits.
For each pipeline depth (1, 4, 16, 64, 256, and 512), in this order, the harness runs the following steps:
-
Flush every server with
redis-cli flushall, retrying up to 10 times, 1 second apart, until all servers answer. This is also the only wait for the servers to start. -
Record the CPU times of the servers and the clients' running totals.
-
Start one client per server, all at once:
memtier_benchmark -4 -p <port> --ratio=1:0 --test-time=10 --hide-histogram -t 1 --pipeline=<depth>The
--ratio=1:0option sends SET operations only. -
Wait for every client to exit, read the SET results from each client's summary table, and record the CPU times again.
Design History
An earlier version of the harness also ran each pipeline depth with two and four
memtier_benchmarkmemtier_benchmarkAn open-source load generator for Redis and Memcached, developed by Redis Ltd. It runs many concurrent connections over one or more threads, sends a configurable mix of SET and GET requests, and reports operations per second, latency averages, and latency percentiles per request type.memtier_benchmark on GitHubGlossary threads per client, which produced 18 rows instead of six.
We cut it to one thread in August 2024: with one server and client pair per
CPU, more threads do not improve performance. A few published results still
come from the earlier version, for example
aws/c6g.16xlarge.
For each pipeline depth, the Navigator keeps the row with the highest rps,
regardless of the thread count.
References
See References for the terms specific to this image.