Mechanism◆Article

Bundling every version of a document into a single blob and applying a good compression algorithm should wipe out most of the redundant text.

The core insight: rather than compress cleverly per-column, take the full text of every prior version as a JSON array of strings and compress the whole thing with zlib or zstd. ✦ AI generated

The author (as spoken to GPT-Live) · Simon Willison's Weblog · 2026-08-09 · original ↗

Share card ↓Export .md Export .txt

compression would work really well, right? If you Bundle all of those different um Every every version of this document all the way back to the start if you were to apply a good compression algorithm to them that should basically wipe out huge amounts of the redundant text.

Read full article ↗excerpt · fair-use quotation

✳ zlib ✳ zstd

Around this claim

In practice · 1

The experimental prototype works extremely well: 1,000 simulated revisions of a document compressed from 20.4 MB of raw revision text to 80.3 KB as a Zstandard-compressed JSON array.The author (on GPT-5.6 Sol Pro output) · Simon Willison's Weblog · conf 80%

This moment responds to

explains mechanism → Storing revision histories in relational databases is perennially difficult, and column-per-version approaches waste enormous space on long documents.The author (as spoken to GPT-Live) · Simon Willison's Weblog supports → The scheme is a BLOB history column holding a compressed JSON array of all previous documents, plus an uncompressed parallel JSON array of timestamps.The author (as spoken to GPT-Live) · Simon Willison's Weblog