Web Server
This benchmark measures how fast one binservebinserveAn open-source static web server written in Rust, distributed as a single binary and configured with one JSON file. It maps URL routes to files and directories, can keep files cached in memory, and can also serve encrypted connections and list directory contents.binserve on GitHubGlossary process serves static files
over HTTPHTTP: Hypertext Transfer ProtocolThe request-response protocol of the web, in which a client asks a server for a resource by URL and gets back a status code, headers, and a body. HTTPS is HTTP over an encrypted connection.MDNGlossary, with wrkwrkAn open-source HTTP benchmarking tool written in C. It generates load from a few threads, each handling many open connections through an event loop, and reports latency and requests-per-second statistics. Lua scripts can customize the requests it sends and process the responses.wrk on GitHubGlossary as the load generator. Server and client run in the same
container and talk over loopback. It sweeps five file sizes, from 1 KiB to
512 KiB, and the number of wrk threads and connections, which scale with the
number of CPUsCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary. Each of the 69 configurations runs once for 10 seconds.
It reports the requests per second and the average latency of each configuration, together with the CPU time the server and the client used.
Spare Cores Navigator publishes
requests per second, throughput in bytes per second, and latency for each file
size and number of connections per vCPUvCPU: virtual CPUA CPU of a virtual machine, the unit in which cloud vendors sell compute. It is usually one hardware thread of a physical core with SMT, so two vCPUs may share one physical core.WikipediaGlossary. It also publishes extrapolated
requests per second and throughput, which estimate how fast binserve would
serve if wrk did not share its CPUs; see Results.
Purpose
This benchmark shows how fast static files can be served over HTTP on a server type, and how throughput and latency change with the file size and the number of concurrent connections.
We serve the files with binserve because it is a single binary that needs no
tuning. Widely used web servers, such as nginxnginxAn open-source web server, also used as a reverse proxy, load balancer, and HTTP cache. It handles many connections in a few worker processes, each running an event loop, and has many settings for workers, buffers, caching, and timeouts that are usually tuned per deployment.nginx docsGlossary, have many settings, some of
them best set per server type, which would make a like-for-like comparison
across server types hard.
The web server and the load generator run side by side in the same
container: wrk sends its requests to binserve at localhost:8080, over
the loopback interface. Loopback is a virtual network interface that the
operating systemOS: operating systemThe software that manages a computer's hardware and runs programs on it, such as Linux or Windows. A cloud server runs the OS of the image it was started from.Glossary provides for traffic between programs on the same machine,
so the requests never reach a network card or leave the server.
This setup needs only the server under test, not a separate client machine,
which reduces provisioning and benchmark costs. It also keeps network latency
out of the results. However, wrk uses CPU time that binserve could
otherwise use. This benchmark records both processes' CPU time so the
Navigator can estimate binserve performance without the client's CPU cost;
see Results.
Limitations
These results compare static file serving by one binserve process on one
host, not the capacity of a tuned production web server. The main limitations
are the following:
- Shared CPUs:
binserveandwrkshare the same CPUs, andwrkruns up to four threads per CPU, sorpsis not the maximum a dedicated server would reach. The Navigator also publishes CPU-time-adjusted scores. - Large Servers: This benchmark does not scale well on servers with 100 or
more vCPUs because of a listener bottleneck in the single
binserveprocess. - Static Files Over HTTP: This benchmark serves five static
files, filled with zero bytes, from
binserve's in-memory file cache over plain HTTP. It does not measure dynamic content or HTTPSHTTPS: Hypertext Transfer Protocol SecureHTTP, the protocol of the web, encrypted with TLS. It protects requests and responses from eavesdropping and tampering, and its certificates let clients verify the identity of the server.WikipediaGlossary. - Loopback Only:
wrkandbinserveexchange requests over the loopback interface of the same machine, so no traffic crosses a network card or a real network. Results do not include network throughput or latency, and a server type's network performance does not affect them. - Untuned Server:
binserveruns with the settings inbinserve.jsonand its defaults otherwise. Results do not show how a tuned web server, such as nginxnginxAn open-source web server, also used as a reverse proxy, load balancer, and HTTP cache. It handles many connections in a few worker processes, each running an event loop, and has many settings for workers, buffers, caching, and timeouts that are usually tuned per deployment.nginx docsGlossary, would perform on the same server type. - Single, Short Runs: Each configuration runs once for 10 seconds, with no warmup and no repetitions.
- CPU Count: The thread and connection counts follow the CPUs the
container may run on. Docker's
--cpuset-cpuschanges them, but--cpusdoes not: with--cpus, this benchmark still uses the host's CPU count. - Error Checks:
failedcounts only responses with a status other than 2xx or 3xx, andwrk's connect, read, write, and timeout errors are not recorded. Ifwrk's output cannot be parsed, this benchmark prints the error to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary, writes the row with anrpsof 0, and still exits with code 0, as if the run had succeeded. The Navigator publishes such a row only if every thread count in its group returned zero.
Usage
Run the image via Docker:
docker run --rm --privileged ghcr.io/sparecores/benchmark-web:main
The harnessharnessThe code that runs a benchmark tool. It prepares the inputs, starts the tool with fixed settings, often several times or in parallel, checks and collects the tool's output, and prints the results in one format. Its settings, not the tool's defaults, decide what the benchmark measures.WikipediaGlossary runs binserve and wrk at the highest scheduling priority
(nice -n -20niceA Unix command that runs a program with a changed scheduling priority, from -20 (highest) to 19 (lowest). Raising a process's priority above the default needs the CAP_SYS_NICE capability, which root has.man pageGlossary), which needs the CAP_SYS_NICE capability that --privileged
grants. Without it, they run at normal priority. The harness also raises its
open-file limit to 1,048,576 before starting them.
Production runs the container the same way and also sets host networking
(--network host). With host networking, binserve listens on port 8080 of
the host's loopback interface, so that port must be free. If it is not,
binserve cannot listen on it, and this benchmark usually stops with
Couldn't start server and exit code 1. Without host networking, the traffic
stays on loopback inside the container.
The container entrypointcontainer entrypointThe executable that a container image runs when a container starts, set by the ENTRYPOINT instruction of its Dockerfile. The image's default command (CMD), or the arguments given after the image name in docker run, are passed to it as arguments, so an entrypoint can wrap any command, for example to set up the environment or record resource usage first. The docker run --entrypoint option replaces it.Dockerfile referenceGlossary runs this benchmark under resource-trackerresource-trackerAn open-source Spare Cores tool that runs a command and samples resource usage while it runs, such as CPU, memory, disk, network, GPU, and GPU memory use. It can optionally stream the samples to the Spare Cores Sentinel platform, so that a run can be checked to see its aggregated and detailed resource utilization, e.g., whether it used all CPU or GPU cores as intended while running a Navigator benchmark workload.resource-tracker on GitHubGlossary, which
samples resource usage during the run. Its data is not published.
Measurement takes about 12 minutes: 69 configurations of 10 seconds each, plus
server startup. A production run on an aws/m5.large took 11 minutes 42 seconds
(run metadata).
The run metadata also records the tool versions in its version field, as one
string, for example binserve 0.2.1, wrk 4.2.0. The image builds binserve
from commit 1e32f94 and wrk from version 4.2.0.
Settings
This benchmark reads no environment variables or command-line options. The
file sizes, thread and connection counts, and run time are fixed in
benchmark.py; see Workload. To change the
thread and connection counts, limit the container's CPUs with
--cpuset-cpus.
Results
This benchmark prints a comma-separated table to stdoutstdoutThe standard output stream of a process, one of the three standard streams with stdin and stderr. Programs write their results to it; a shell can redirect it to a file or another program, such as > results.jsonl, while logs on stderr still reach the terminal.WikipediaGlossary: a header row with
quoted column names, then one row per configuration.
"size","threads","connections","rps","latency","failed","server_usr","server_sys","client_usr","client_sys"
The columns mean the following:
sizeis the label of the served file in KiB, such as"1k".threadsandconnectionsare thewrkthread count and the number of open connections.rpsis the requests per second thatwrkreports, as an integer.latencyis the average latency fromwrk's thread statistics in seconds.failedis the number of responses with a status other than 2xx or 3xx.server_usrandserver_sysare the user and system CPU seconds thatbinserveused during the configuration.client_usrandclient_sysare the user and system CPU seconds thatwrkused during the configuration.
For example, on an aws/m5.large (2 vCPUs), serving the 1k file with two
threads, rps rises from 43,808 with two connections to 64,606 with 32
connections, while latency rises from 0.05 ms to 0.55 ms
(production output).
The Navigator groups the rows by file size and by connections per vCPU
(connections divided by the server's vCPU count), so that servers with
different vCPU counts can be compared. In each group, it keeps the row with the
highest rps, whatever its thread count, and publishes the following five
scores from it:
static_web:rps:rpsin requests per second.static_web:rps-extrapolated:rpsmultiplied by the ratio of all CPU time (server and client) to the server's CPU time in requests per second.static_web:throughputandstatic_web:throughput-extrapolated: the two scores above multiplied by the file size in bytes per second.static_web:latency:latencyof the same row in seconds; lower is better.
In the aws/m5.large example, for the 1k file at 16 connections per vCPU,
the row with two threads (64,606 requests/s) is kept over those with
four and eight threads. Note that the published latency is that of the
highest-rps row, not the lowest latency in the group. The Navigator's Web UI
shows these scores with charts on each server type's page, for example
aws/g5g.xlarge.
Workload
This benchmark runs one workload: a sweep of file sizes, wrk threads, and
connections against one binserve process. benchmark.py
first runs the following steps:
- Create five files in
/usr/local/binserve/data/, with sizes of 1, 16, 64, 256, and 512 KiB, filled with zero bytes. - Start one
binserveprocess withbinserve.json, which listens on127.0.0.1:8080, routes/1kto/512kto the files, and turns on the in-memory file cache and turns off logging. This process serves every configuration. - Wait for
binserveto be ready: send aHEAD /1krequest up to 10 times, 1 second apart, until it answers200 OK. If it never does, printCouldn't start serverand exit with code 1.
Then, for each file size from the smallest to the largest, the harness runs
wrk once for each combination of the following counts:
- Threads: 1, 2, or 4 per CPU.
- Connections: 1, 2, 4, 8, 16, or 32 per CPU, but at most 16 per CPU for
the 256 KiB and 512 KiB files. A comment in
benchmark.pysays smaller files likely need more connections to saturate the machine.
Combinations with fewer connections than threads are skipped. This leaves 15 configurations for each of the three smaller files and 12 for each of the two larger ones, 69 in all, whatever the CPU count. Each configuration runs the following command, one after another, with no warmup:
wrk -d 10 -t <threads> -c <connections> http://localhost:8080/<size>
Before and after each run, the harness records the CPU times of binserve and
of wrk, which give the CPU columns of the row.
References
Common terms, such as binserve, wrk, and HTTP, are defined in the
glossary of the Spare Cores docs.