Video · 2026-06-09 · 51m · 6 moments

Why Bigger Context Windows Don't Solve the RAG Problem

✦ AI generated

timeline · colored by role

Claim◆

Sphere built an internal system called TRAM (Tax Review and Assessment Model) that lets human tax experts move nearly 100x faster through tax-determination review than a fully manual process, while also reducing errors.

Alex describes TRAM as an AI system built to supercharge tax experts, claiming it moves them almost two orders of magnitude faster than the traditional manual approach with fewer errors.

transcript

Alex Boucott: What we eventually built is what we call TRAM, which is the tax review and assessment model, which is like a system of a few different things... essentially, it its job is to supercharge our tax experts. So, what we found is that TRAM allows our internal tax experts to move almost two orders of magnitude faster through this process with less errors than the traditional just fully human, you know, focused approach.

Mechanism

Naively chunking documents by a fixed number of characters loses important context, whereas semantically chunking legal documents along their natural structural boundaries (sections, subsections, bullets) while retaining hierarchy preserves accuracy that a naive approach would leave on the table.

Alex explains that Sphere invested heavily in semantic, structure-aware chunking of legal documents rather than naive character-count chunking, because the naive approach sacrifices retrieval accuracy.

transcript

Alex Boucott: There's a naive way to do that, which is just like um every n characters you chop and then you create a new section. And that's obviously not ideal cuz you you lose very relevant context... what we try and do is our pipeline semantically chunk things into essentially sensible chunks that that cut at normal places. And then we still retain the hierarchy of where that chunk came from so that we can reproduce it later.

provides context · 1

Data

Combining sparse (full-text/TF-IDF style) search with dense semantic embeddings improved citation retrieval accuracy over using dense embeddings alone, because legal terminology and exact citation wording benefit from exact-term matching that dense embeddings can miss.

Alex describes how adding sparse retrieval (via Pinecone's TF-IDF-like vocabulary index) back alongside dense embeddings measurably increased citation accuracy in Sphere's evals.

transcript

Alex Boucott: There are times especially with certain when it comes to citations um and and pulling out certain terms from passages that come from the dense embeddings where you also want to search sparse where you want to do you know a full text search of certain uh words and certain terms and pull those in as well... and what we saw was a pretty good increase in accuracy on the citation side.

Claim◆

For accuracy-critical, citation-sensitive domains like tax law, agents grepping over a file system aren't yet accurate enough to replace a purpose-built retrieval system — RAG isn't dead for this use case, even though it may be for others.

Alex Boucott argues that while 'RAG is dead' may hold for some use cases, Sphere's need for exact, verifiable citations means agent-based grepping over documents still can't match a dedicated retrieval pipeline's accuracy.

transcript

Alex Boucott: I think for us and or at least for this particular problem, because we are so sensitive to accuracy and we're so sensitive to the exact right citation... as of today, I don't think, you know, agents are just searching over the file system, grepping over it is at a point where we could switch over and and not lose accuracy.

explains mechanism · 2extends · 1provides context · 1

Mechanism◆

Using accumulated human tax-expert feedback on model errors as ground truth and reward signal made OpenAI's reinforcement fine-tuning (RFT) especially effective for Sphere's tax-determination task, since it targeted precisely the hard cases the model had previously missed.

Alex explains that Sphere's existing pipeline of tax-expert feedback on model mistakes provided ideal ground truth and signal for OpenAI's RFT alpha program, yielding accuracy improvements now used in production.

transcript

Alex Boucott: What we had that was very useful was feedback from our human tax experts every time the model T RAM had gotten something wrong on a determination... we had the ground truth, we had signal, and we knew that these were hard problems that the model had missed previously. And so, that was a really good recipe for RFT, and we saw uh improvements with, uh, during the alpha program with with OpenAI on RFT.

Mechanism

Larger context windows and better reasoning models don't eliminate the need for precise retrieval, but they do let Sphere be less precise upstream and expand retrieved passages more aggressively, since reasoning models handle more context without the degradation earlier non-reasoning models showed at their context limits.

Alex says growing context windows and improved reasoning models let Sphere feed in more, less-precisely-selected passages without the accuracy collapse seen in earlier models, though they still don't fill the context window to its max.

transcript

Alex Boucott: I think reasoning models are much more capable of reasoning over their full context, whereas non-reasoning models, yeah, you got real degradation... a big unlock was models where we could give it more, where maybe we could be a little less precise on the retrieval portion and expand expand these passages a little more aggressively.

Highlight slides

TRAM: AI-Powered Tax Review at Sphere✦ from: Sphere built an internal system called TRAM (Tax Review and Assessment Model) that lets human tax experts move nearly 100x faster through tax-determination review than a fully manual process, while also reducing errors.Nearly 100x Faster, Fewer Errors✦ from: Sphere built an internal system called TRAM (Tax Review and Assessment Model) that lets human tax experts move nearly 100x faster through tax-determination review than a fully manual process, while also reducing errors.RAG Isn't Dead for Tax Law✦ from: For accuracy-critical, citation-sensitive domains like tax law, agents grepping over a file system aren't yet accurate enough to replace a purpose-built retrieval system — RAG isn't dead for this use case, even though it may be for others.Grepping vs. Dedicated Retrieval✦ from: For accuracy-critical, citation-sensitive domains like tax law, agents grepping over a file system aren't yet accurate enough to replace a purpose-built retrieval system — RAG isn't dead for this use case, even though it may be for others.Why RFT Worked for Sphere's Tax Model✦ from: Using accumulated human tax-expert feedback on model errors as ground truth and reward signal made OpenAI's reinforcement fine-tuning (RFT) especially effective for Sphere's tax-determination task, since it targeted precisely the hard cases the model had previously missed.From Alpha Test to Production✦ from: Using accumulated human tax-expert feedback on model errors as ground truth and reward signal made OpenAI's reinforcement fine-tuning (RFT) especially effective for Sphere's tax-determination task, since it targeted precisely the hard cases the model had previously missed.

Related episodes

Video · Lenny's Podcast

The rise of taste, human authenticity and judgment in an AI world | Adam Mosseri (Head of IG)

→

Video · a16z Podcast

Garry Tan: New Rules for Founders

→

Video · Syntax

I Built an LLM From Scratch

→