Naively chunking documents by a fixed number of characters loses important context, whereas semantically chunking legal documents along their natural structural boundaries (sections, subsections, bullets) while retaining hierarchy preserves accuracy that a naive approach would leave on the table.
Alex explains that Sphere invested heavily in semantic, structure-aware chunking of legal documents rather than naive character-count chunking, because the naive approach sacrifices retrieval accuracy. ✦ AI generated
Alex Boucott · The TWIML AI Podcast · 2026-06-09 · original ↗
starts at this moment · 17:08
“How did you approach that? Was it just obvious that you know, hey, we're going to do this based on sections... or did you like iterate on that for a while?”
There's a naive way to do that, which is just like um every n characters you chop and then you create a new section. And that's obviously not ideal cuz you you lose very relevant context... what we try and do is our pipeline semantically chunk things into essentially sensible chunks that that cut at normal places. And then we still retain the hierarchy of where that chunk came from so that we can reproduce it later.
verbatim transcript · starts at 17:08
17:08there you know, we can't many of these documents are are very long. So we we can't just like take the the document and, you know, create an embedding for it and and store that in a vector database or or even necessarily with like a more with like a TF-IDF um uh like full text search databases. You you might not want to do that either. So, what we do is we break up into sections,
17:31smaller sections. And there's a naive way to do that, which is just like um every n characters you chop and then you create a new section. And that's obviously not ideal cuz you you lose very relevant context. And again, these are legal documents, so they're well-structured typically, as long as it's not an image. Um and so, they you know, they come in sections and then subsections and bullets. And so,
17:57what we try and do is our pipeline semantically chunk things into essentially sensible chunks that that cut at normal places. And then we still retain the hierarchy of where that chunk came from so that we can reproduce it later. Um and then we also store like metadata, of course, and things about where this document came from, the like the root document. Um and then eventually we we
18:21have these these text chunks and we embed those both dense and sparse. Um and we store them in a vector database. And that's eventually what we'll we'll then query over when we go to actually make a determination. But I think it is you know, we don't I could probably talk for the next 60 minutes about kind of this process of of chunking. I think we spent a lot of time there and it's it's
18:44a very important part of this process. I think if you do a naive implementation, you leave a lot of accuracy on the table, essentially. >> Yeah, I wouldn't mind having you dig into some of the work that you've done to kind of assess the the lift on the semantic chunking. Um and what you've seen there. I think you know, as you alluded to, like a lot of folks will pull a rag library off the