Recursive context buckets organized in a red-black tree structure allow AI agents to access billions of tokens across thousands of context windows in just two LLM calls.
Lindy built a system of recursive 'context buckets' — sub-agents that each hold up to 200K tokens of context. These nest inside each other like Russian dolls, but organized in a 100-ary red-black tree so that accessing 10,000 context buckets (2 billion tokens) requires only two LLM calls. This enables agents to remember everything perfectly at all times. ✦ AI generated
Flo Crivello · The Cognitive Revolution · 2026-08-10 · original ↗
starts at this moment · 28:00
“How are you managing caching and cost in general given all the tokens getting processed?”
We call them context buckets... we went one step further and we were like, hey, what if we could have recursive context buckets? What if context buckets could contain other context buckets?... What you do instead is that you have context buckets contain multiple context packets. Okay? And and and and so you end up with a tree... So red black trees the canonical implementation of red trees is on binary trees which is just a tree where each nodes got two children. We went for it's called a centenary tree. It's literally just each node's got 100 nodes. So the the node below that's going to have like 10,000 nodes, right? And so literally with two jumps, you can have 10,000 context buckets and you can access like all the context in the universe... that is what leads to like those really surprising behaviors from those AI agents where you ask them any question and they remember everything perfectly all the time.
verbatim transcript · starts at 28:00
28:00call them context and the the the way it started was if an agent calls an action that returns too much context like some some actions some MCPS in particular retrieve like a 100,000 tokens or something okay you don't want to send that to the to the agent is going to get confused. Okay. So what you do is that you expose that as as a as a summary of the context bucket
28:26and it's a sub agent which contains the entire bucket. It's like hey this action return too much so I'm here you know to stand in for what the actually return roughly speaking this is what it contains. Okay. And then the agent can enter into a conversation with the sub agent which itself has its own caching and can manipulate the context using Unix utilities. So right here you're
28:47saving a lot. you're saving a lot of money and it goes quite fast. Then we went one step further and we were like, hey, what if we could have recursive context brackets? What if context brackets could contain other context brackets? And what if compaction because obviously we have compaction was powered by such recursive context brackets and and by that I mean like now when we compact so the conversation goes it passes
29:12threshold. I think right now it's like 200,000 tokens, but like we keep we keep tweaking it and at some point we're like, okay, we're going to compact. We compact and then the compaction sends sends all of that stuff into a context. So now the agent can query that context. So it's not like every compaction is always lassy by nature and I think compaction operates under the faulty
29:34assumption that you never need access to ground truth which is is false. You you at some point do need access to ground truth. So with that technique you have access to ground truth and then we went step further which is so you you have that context packet which is comp the compaction of the of the previous conversation. Your conversation keeps going keeps going keeps going you need
29:52to summarize again. What you do is that you take all of that including this context bucket right here and you compact it into a new context bucket. So now you have a context bucket containing a context bucket and in the end so you do that and so you have this emerging property where it's like you can have the agent access any point of any infinite number of tokens
30:12at arbitrary levels of of granularity. Okay, the problem is that if you if you have a computer science background like this gives you like what's called like N complexity, right? So it's like if it wants to access like nine context buckets ago, it's got to go through nine layers of sub aents and that's really slow and really expensive. And so what what we ended up doing is maybe it's
30:30getting too technical, but we never >> Okay. Well, have you heard of AVL trees? >> No, but I'm all ears. >> Have you have you heard of red black trees? Um there is this >> I went to this I went to the computer science school of hard knock so it wasn't uh formal training for me. I was actually glad to to have AVL and black trees because I was like oh my god you