Semantic retrieval using embeddings solves the proxy problem: dual-encoder models embed users and posts into a shared space so that related meanings—not surface keywords—position items nearby, and retrieval becomes a fast nearest-neighbor search over a precomputed index.

An embedding is a list of numbers that positions an item as a point in a high-dimensional space, arranged so that related items land near one another. For example, a post about fixing a leaking tap and a post about reducing water waste sit close together even when they share no keywords, because their meanings are related... One encoder converts a user, along with their profile and recent activity, into a point in the space. A second encoder converts each post into a point in the same space. Since the two encoders operate independently, a platform can compute every post embedding in advance and store it in an index. At request time, it computes the user embedding and runs a nearest-neighbor search to find the closest posts, which keeps retrieval fast across an enormous corpus.

Read full article ↗excerpt · fair-use quotation