Compression
Running the Script
docker run --rm --privileged ghcr.io/sparecores/benchmark:main python /usr/local/bin/compress.py
In production, the median run takes ~10.7 minutes. See
Run Duration for how it was measured, and
Production Setup for how production runs differ
from a plain docker run.
Workload
compress.py measures how well and how fast Python
compression libraries handle one input: dickens, a text file of ~10 MB
from the Silesia corpusSilesia corpusA set of 12 files, ~211 MB in total, compiled by Sebastian Deorowicz at the Silesian University of Technology to test lossless compression on a mix of data types: text, executables, databases, images, and XML. Its dickens file holds the collected works of Charles Dickens, ~10 MB of English text.Silesia corpusGlossary. The script reads it into memory once.
It runs every compressor at the following levels and extra arguments:
| Compressor | Python library | Levels | Extra arguments |
|---|---|---|---|
brotlibrotliA lossless compression format by Google, standardized in RFC 7932. It combines LZ77-style matching, Huffman coding, and context modeling with a built-in dictionary of common web text, and is widely used to compress HTTP responses. Its quality levels range from 0 (fastest) to 11 (smallest output).RFC 7932Glossary | brotli | 0, 4, 8, 11 | None |
gzipgzipA lossless file compressor of the GNU Project, first released in 1992, and its file format, specified in RFC 1952. It uses the DEFLATE algorithm, which combines LZ77 matching and Huffman coding, and is the most widely supported compression format on Unix-like systems and the web. Its levels range from 1 (fastest) to 9 (smallest output).GNU gzipGlossary | gzip | 1, 5, 9 | None |
bzip2bzip2A lossless file compressor by Julian Seward, first released in 1996. It sorts each block of data with the Burrows-Wheeler transform, then encodes it with move-to-front and Huffman coding. It usually compresses better than gzip but runs slower; its levels 1 to 9 set the block size from 100 to 900 kB.bzip2Glossary | bz2 | 1, 5, 9 | None |
lzma | lzma | 1, 5, 9 | None |
zstd | pyzstd | 1, 7, 14, 22 | None |
bzip3bzip3A lossless compressor by Kamila Szewczyk that, like bzip2, is built on the Burrows-Wheeler transform, combined with context modeling. It typically compresses text better than bzip2, and its configurable block size trades memory use for compression ratio.bzip3 on GitHubGlossary | bz3 | No level | block_size 1 MiB (default) and 64 MiB |
zpaqzpaqA lossless compressor and archive format by Matt Mahoney, based on context mixing, which predicts each bit from many models and combines their predictions. It reaches very high compression ratios, but compresses and decompresses slowly, at about the same speed in both directions. Its methods range from 1 (fastest) to 5 (smallest output).ZPAQGlossary | pyzpaq | 1, 3, 5 | None |
lz4lz4A lossless compression algorithm and format by Yann Collet from the LZ77 family, built for speed rather than compression ratio. It compresses and especially decompresses much faster than gzip, at a lower ratio; its higher levels use a slower high-compression mode that keeps decompression fast.LZ4Glossary | lz4.frame | 1, 6, 12, 16 | block_size 64 KiB (default), 1 MiB, and 4 MiB |
Each combination runs with one thread and with NUMCPUS threads, where
NUMCPUS is the smaller of the physical core count and the number of CPUsCPU: Central Processing UnitThe general-purpose processor of a computer, which runs the operating system and programs. A CPU has one or more physical cores, and with SMT each core runs two or more hardware threads.WikipediaGlossary the
process may run on. On a server with one core, only the single-thread case
runs.
How a Measurement Works
Each combination runs in its own worker process, so the kernel's out-of-memory killerout-of-memory killerThe Linux kernel mechanism that frees memory when the system, or a cgroup, runs out of it and cannot reclaim enough. It picks a process by a badness score, mostly based on its memory use, and kills it. Container runtimes rely on it to enforce memory limits.Linux kernel docsGlossary stops only that measurement. A measurement works as follows:
- Compresses the input once on one thread to compute the ratio, then decompresses it and checks that the result matches the input byte for byte.
- Starts one process per thread, each compressing the full input independently. The processes wait for a shared start signal, sent after 1 second.
- Takes the mean of the per-process times as the elapsed time, and computes throughput as the input size times the thread count, divided by the elapsed time.
- Repeats steps 2 and 3 once more and keeps the faster run.
- Measures decompression the same way.
"Threads" here means parallel, independent processes that each compress the same data, not one multithreaded compressor working on shared data.
Output
The compress.py script prints one indented JSONJSON: JavaScript Object NotationA text format for structured data, built from objects (key-value pairs), arrays, strings, numbers, booleans, and null. Most languages and tools read and write it, which makes it a common format for API responses and benchmark output.JSONGlossary object,
{<compressor>: {<level>: [<result>, ...]}}. Levels are string keys, and
bzip3bzip3A lossless compressor by Kamila Szewczyk that, like bzip2, is built on the Burrows-Wheeler transform, combined with context modeling. It typically compresses text better than bzip2, and its configurable block size trades memory use for compression ratio.bzip3 on GitHubGlossary, which has no level, uses the key "null". Each result has the
following fields:
threads: the number of parallel processes.extra_args: the extra arguments, such asblock_sizein bytes forbzip3andlz4lz4A lossless compression algorithm and format by Yann Collet from the LZ77 family, built for speed rather than compression ratio. It compresses and especially decompresses much faster than gzip, at a lower ratio; its higher levels use a slower high-compression mode that keeps decompression fast.LZ4Glossary.ratio: compressed size divided by original size, as a percentage; lower is better.compressanddecompress: bytes of uncompressed data per second, summed over all processes.
A measurement that fails, for example because its worker process was killed,
has ratio, compress, and decompress set to null, and its traceback goes
to stderrstderrThe standard error stream of a process, one of the three standard streams with stdin and stdout. Programs write logs, progress, and error messages to it, so that they stay separate from the output on stdout when the output is redirected to a file or another program.WikipediaGlossary. Production runs on small servers show such failures.
Compression Input
The compression benchmark uses only dickens, a text file of the Silesia
corpusSilesia corpusA set of 12 files, ~211 MB in total, compiled by Sebastian Deorowicz at the Silesian University of Technology to test lossless compression on a mix of data types: text, executables, databases, images, and XML. Its dickens file holds the collected works of Charles Dickens, ~10 MB of English text.Silesia corpusGlossary, although the image ships the whole corpus. We needed one large-enough,
common text file to compare the algorithms, and there are no plans to add
other corpus files. The results show how the compressors handle text, not
other data types.
CPU Count
The multi-thread case starts one process per physical core, not per logical CPU (see Workload). On a server with SMTSMT: Simultaneous MultithreadingA CPU technique that runs two (or more) hardware threads on one physical core, sharing its execution units, e.g. Intel Hyper-Threading. It raises throughput when a thread stalls, but the threads compete for the core. Cloud vendors usually sell each thread as a vCPU.WikipediaGlossary (Simultaneous Multithreading), its results therefore do not use every logical CPU.
Library Versions
The Python compression libraries are installed unpinned from PyPIPyPI: Python Package IndexThe official public repository of third-party Python packages, run by the Python Software Foundation. Tools such as pip download and install packages from it by default.PyPIGlossary (Python
Package Index) at build time, except pyzpaq, which is pinned to one commit.
A rebuilt image can measure newer library versions.