ATRIUMsearch → argument graph
Article · 2026-08-09 · 5 moments

SQLite compressed text-history prototypes

Research: SQLite compressed text-history prototypes I'm perennially interested in options for storing revision histories in relational databases. While out on a dog walk I had a new idea: how about taking the full text of every prior version in a big JSON array of strings and then applying zlib or zstd compression to the whole thing? Surely that would compress really well due to all of the repeated strings. The new GPT‑Live voice mode in the ChatGPT iPhone app has got really good, so I ✦ AI generated

01
Mechanism

Bundling every version of a document into a single blob and applying a good compression algorithm should wipe out most of the redundant text.

The core insight: rather than compress cleverly per-column, take the full text of every prior version as a JSON array of strings and compress the whole thing with zlib or zstd.

transcript

The author (as spoken to GPT-Live): compression would work really well, right? If you Bundle all of those different um Every every version of this document all the way back to the start if you were to apply a good compression algorithm to them that should basically wipe out huge amounts of the redundant text.

gives example · 1

02
Context

Storing revision histories in relational databases is perennially difficult, and column-per-version approaches waste enormous space on long documents.

The author sets up the problem: naive storage of a row per previous version means every edit to a large document adds another full copy to the database.

transcript

The author (as spoken to GPT-Live): I've built these kinds of systems in the past, and it's always difficult to come up with a efficient way to do this. Like the easiest way is you have a row for every previous copy of the previous previous value of the string. But if it's a long document Like20 kilobytes of data, that means that every single edit adds another 20 kilobytes of data to the database, right.

explains mechanism · 2gives example · 1

03
Data

The experimental prototype works extremely well: 1,000 simulated revisions of a document compressed from 20.4 MB of raw revision text to 80.3 KB as a Zstandard-compressed JSON array.

GPT-5.6 Sol Pro built the prototype (38 minutes of work), and the results confirm the approach: a roughly 250x compression ratio.

transcript

The author (on GPT-5.6 Sol Pro output): The approach works really well! 1,000 simulated revisions to a document resulted in 20.4 MB of raw revision text that compressed to 80.3 KB as Zstandard-compressed JSON array.

04
Mechanism

The scheme is a BLOB history column holding a compressed JSON array of all previous documents, plus an uncompressed parallel JSON array of timestamps.

The concrete design: a single table with a compressed JSON text-array BLOB and an uncompressed JSON array of Unix integer timestamps.

transcript

The author (as spoken to GPT-Live): There is a history column on the single on this uh uh table and it's a blob, it's a BLOB so it stores binary data and then you just stick in there a Zlib or maybe even ZSTD um compressed JSON text array of all of the previous documents, and so you probably have two columns, right? You'd have a column that's this magic JSON array of text You have a second column which is a JSON array of timestamps and that doesn't need to be compressed at all, right? A timestamp can just be a uh it's an array of integers, right? Unix integers.

gives example · 2supports · 1

05
Prediction

To avoid decompressing and recompressing the entire array on every edit, the history should be split across multiple rows capped at 128 revisions or 3MB of uncompressed JSON each.

A scalability refinement: chunking the compressed history into multiple rows bounds the cost of each subsequent edit.

transcript

The author (on GPT-5.6 Sol Pro output): To avoid the overhead of decompressing and recompressing the entire array on every edit Sol suggested breaking the history up into multiple rows, with each one containing a maximum of either 128 revisions or 3MB of uncompressed JSON.

Highlight slides
Related episodes