Larger context windows and better reasoning models don't eliminate the need for precise retrieval, but they do let Sphere be less precise upstream and expand retrieved passages more aggressively, since reasoning models handle more context without the degradation earlier non-reasoning models showed at their context limits.
Alex says growing context windows and improved reasoning models let Sphere feed in more, less-precisely-selected passages without the accuracy collapse seen in earlier models, though they still don't fill the context window to its max. ✦ AI generated
Alex Boucott · The TWIML AI Podcast · 2026-06-09 · original ↗
starts at this moment · 43:13
“I'm wondering the degree to which context length changes the way you approach the problem... How do you think about the impact of context window?”
I think reasoning models are much more capable of reasoning over their full context, whereas non-reasoning models, yeah, you got real degradation... a big unlock was models where we could give it more, where maybe we could be a little less precise on the retrieval portion and expand expand these passages a little more aggressively.
verbatim transcript · starts at 43:13
43:13>> I think that was actually one of the big reasons why we saw a jump with the release of 01 back in the day. Was I think reasoning models are much more capable of reasoning over their full context, whereas non-reasoning models, yeah, you got real degradation as even if it supported, you know, 128K tokens, uh when you push that limit, it was not, you know, needle in the haystack wasn't
43:39great on those sorts of things. And so I think we saw big improvements there um with reasoning models. And so it's still a balance for us. Uh like we like I kind of mentioned earlier, we don't need to fill up and we don't fill up the context window to its max. But a a big unlock was models where we could give it more, where maybe we could be a
44:00little less precise on the retrieval portion and expand expand these passages a little more aggressively. I think before when context was more limited, you know, we were being very selective on which passages we're feeding in because we we you know, we only had so much we could give it before the model just kind of would throw its its its hands up. And so that that was a big
44:23unlock. So, yeah, we don't push the boundary right on the edge, but I think as reasoning models improve, as the context window gets bigger, again, we won't fill it up all the way, but that's a good sign that the model can handle more more more tokens than we're giving it today. And that means we can be less precise a bit on the retrieval portion and still get the the results that we're
44:45looking for. >> How much time do you spend thinking about trying to reduce token costs either by kind of refactoring from larger models to smaller models or, you know, via other methods? >> LLMs compared to lawyers, like a human, you know, tax lawyers are considerably cheaper even the most expensive LLMs. So, yeah, this isn't and this isn't also something this isn't a process where you know, we're we're pushing through
45:13billions of tokens every >> I guess it helps that you're building a deterministic system and that is the thing that's you know, the kind of the in-line online system as opposed to an LLM inference call. >> Yeah, exactly. We're not cost sensitive and that also means we're all we're not latency sensitive either. So, those it's very nice. Those are two things that we don't even really have to consider very