Every current approach to AI agent memory is fundamentally a hack — essentially a vector database or bolted-on tool — and it's hard for agents to know when to query it, whether it's stale, or whether to trust it over fresh reasoning.
Pullen says memory remains one of the hardest unsolved problems in agentic AI — every current implementation is essentially a hacky external tool, like a vector DB, that agents struggle to query correctly or keep updated, sometimes overriding correct behavior with stale stored information. ✦ AI generated
Alistair Pullen · Machine Learning Street Talk · 2026-07-13 · original ↗
starts at this moment · 45:26
“what are your thoughts on memory”
Yes, memory is very hard to get right. Um, we've tried a bunch of different approaches. Um, fundamentally, I think every approach to memory that exists right now is a bit of a hack, right? It's like a tool. Um, and it's in many cases like a vector DB or like an embedded version of some tidbit of knowledge, but they're very hard for the agents to know when to query.
verbatim transcript · starts at 45:26
45:09is I don't like the abstraction you've done I don't like the way you've done this there is going to have to be some sweeping afterwards I think >> yeah what are your thoughts on memory >> very hard to get right >> okay tell me more >> very hard so >> well similar thing actually to the RL because memory is not about the destination, it's about how you got
45:26there. >> Yes, memory is very hard to get right. Um, we've tried a bunch of different approaches. Um, fundamentally, I think every approach to memory that exists right now is a bit of a hack, right? It's like a tool. Um, and it's it's in many cases like a vector DB or like an embedded version of of like some tip bit of knowledge, but they're very hard for
45:48the agents to know when to query. Um, it's also fundamentally quite hard for the agent to know whether something was useful enough to write to memory. Um, and it's it's it's also difficult um to to keep these things up to date. We've had many situ situations where uh an agent's been doing a trajectory um and when it's been doing something that was genuinely the right thing to do um it's
46:19like used its memory and it's like the memoryy's been old and out of date and then the agent's like oh well the memory says you should do it this way and it changes its tack and there you're like an engineer you're like no please don't do that. There are things that we're looking at internally with regard to um like continue continual learning um and stuff like that to try to avoid memory
46:38being a tool and for it to just something that for it to be something that's just in the latent space with the model. Um that is also very hard. Um but I think it is a more intuitive and elegant solution to it just being a tool. It's also a tool that's very hard to get right during RL because it is a huge surface area for foot gunnery uh in
46:59terms of reward hacking in terms of leakage in terms of it being able to quer something query something from the future that it shouldn't have access to yet despite all the guardrails that you can put in place. It's just hard. >> Yeah, exactly. And and in a sense this this is another area for AI psychosis because I've written a memory CLI and um I I would almost argue that now you
47:18don't even need um you know vector databases and so on. you can just have an inverted index, you know, just a SQL light because the the models are so good at asking in different directions, you know. So, yeah, there is a huge problem. It needs to know to retrieve that. That's a big one, but but it it actually works, but it only works for me because it creates this fractionated spaghetti