ContextArticle

Open datasets like The Stack v3 materially raise the floor for every lab that wants to build competitive code models without relying on closed ecosystems, and open weights are strategically important against calls to restrict distillation.

Multiple high-signal posts pushed back on attempts to sharply separate internet-scale pretraining from output-level distillation. The subtext is that open datasets like The Stack v3 materially raise the floor for every lab that wants to build competitive code models without relying on closed ecosystems, and open weights are strategically important. ✦ AI generated

AINews / Latent.Space · Latent Space · 2026-07-24 · original ↗

Share card ↓Export .md Export .txt

Distillation remains the live ideological fault line: several high-signal posts pushed back on attempts to sharply separate 'internet-scale pretraining' from output-level distillation. @GergelyOrosz compared model inspection via prompting to reverse-engineering a competitor's product, while @SchmidhuberAI emphasized distillation's long lineage. @Suhail argued the practical response is not prohibition but stronger investment in open-weight domestic models, and @garrytan put it more simply: open weights are strategically important. The subtext across these posts is that open datasets like The Stack v3 materially raise the floor for every lab that wants to build competitive code models without relying on closed ecosystems.

Read full article ↗excerpt · fair-use quotation

✳ The Stack v3 ◉ Gergely Orosz ◉ Schmidhuber ◉ Suhail

Around this claim

Evidence · 1

The Stack v3 is the largest open code dataset publicly released, with 114 TB raw, 224M repositories, 44B files, 770 languages, and roughly 5T deduplicated/filtered tokens.AINews / Latent.Space · Latent Space · conf 95%

This moment responds to

supports → Anthropic and OpenAI are consolidating a closed-model duopoly that concentrates power and pricing leverage, and open weight models are the only real counterweight available to startups, schools, and enterprises.Kevin Xu (Interconnected, co-author) · Interconnects rebuts → Open weight models are not truly free to use — the 'free' only refers to R&D, a fixed sunk cost, while COGS from running inference is real, revenue-correlated, and applies just as much to Kimi as to any other model.Ben Thompson · Stratechery gives example → Providing transparency on what is happening with open models is the best way for Interconnects to figure out what is working and to enable the open ecosystem to use open models productively in cost-competitive ways.Interconnects (article author) · Interconnects