Mechanism◆Article
Instead of classifying against a fixed tag vocabulary, let the model freely hallucinate candidate tags, then resolve them to concrete existing tags via vector embeddings over the corpus.
Doug Turnbull's approach: have the model emit imagined tags unconstrained by the existing vocabulary, then use vector embeddings against the corpus to find the concrete tags closest to what it invented. ✦ AI generated
the author · Simon Willison's Weblog · 2026-08-14 · original ↗
Doug Turnbull has a neat solution. Tell the model to output tags without any details of the existing vocabulary, then use vector embeddings against the existing corpus to find the concrete tags that are closest to the ones the model imagined might fit!
Read full article ↗excerpt · fair-use quotation
- ·Model outputs imagined tags, unconstrained by existing vocabulary
- ·Find concrete tags closest to the invented ones
- ·Use vector embeddings against the existing corpus
Around this claim
Mechanism · 2
A tag vocabulary of 1,856 entries is too large to feed to an LLM in one shot as a flat multiple-choice 'which of these tags match' prompt.the author · Simon Willison's Weblog · conf 90%Including an example of the shape of your tags helps the model produce a more useful hallucinated guess.the author · Simon Willison's Weblog · conf 75%
This moment responds to
explains mechanism → Semantic retrieval using embeddings solves the proxy problem: dual-encoder models embed users and posts into a shared space so that related meanings—not surface keywords—position items nearby, and retrieval becomes a fast nearest-neighbor search over a precomputed index.Author (article How to Fight Clickbait: Meta, LinkedIn & YouTube Case Studies) · ByteByteGo Newsletterexplains mechanism → Word2vec's embedding vectors capture real semantic relationships purely from word co-occurrence statistics, so that mathematically subtracting 'man' from 'king' and adding 'woman' lands closest to 'queen', without anyone teaching the model about gender or royalty.CJ · Syntaxexplains mechanism → Word2vec embeddings trained purely on word co-occurrence statistics spontaneously encode concepts like gender and royalty as vector arithmetic, such that king minus man plus woman equals queen.CJ · Syntax