The continual-learning field is split between a 'model' camp that believes real memory requires updating model weights and a 'systems' camp that believes controllable, inspectable memory requires keeping knowledge outside the weights, and each camp views the other's approach as illegitimate.
swyx describes a deep rift in the continual-learning community between researchers who update model weights and those who build external, inspectable memory systems, noting each side dismisses the other's approach as not real learning or not real control. ✦ AI generated
swyx · The Cognitive Revolution · 2026-06-27 · original ↗
starts at this moment · 62:42
The first half is the people that update model weights and the second half is the people are system the more systems people. And I'm just making the observation that these guys don't like each other. Like basically the model people don't view the systems people as legit. And systems people are like, well, the model people, you know, have fun training your model, but you're never ever going to have a memory system that you understand because you're just updating weights.
verbatim transcript · starts at 62:42
62:42machine learning side of the spectrum would be like well we will update model weights and the less less machine learning side will be the other thing. I think this is all comes down to how controllable and interpretable you want your memory to be, right? You're going to recall bad facts. You're going to not be you're going to want to forget things and can you control that? Obviously, the
63:04maximum control is you you don't update model weights and you just control what gets into the system so you can delete and monitor and debug. But uh for full internalization in the model of the things that were learned, you probably do have to train on it. And so that is a whole other discipline that trajectory AI engram all these speakers that are uh and adaption labs these all speakers at
63:29the conference. The first half is the people that update model weights and the second half is the people are system the more systems people. And I'm not making I'm not choosing a side here. I'm just making the observation that these guys don't like each other. Like basically the, you know, the the model people don't view the systems people as legit. And systems people are like, well, the model people,
63:50you know, have fun training your model, but you're never ever going to have an a sort of a memory system that you understand because you're just updating weights. And so it's just continue pre-training or whatever. And I think that's fair. That's a fair discussion. >> That's split. update the model's weights or keep memory in a system you can inspect gets a practical answer once you ask what enterprises actually want.
64:14[music] Quite simply, it just takes one security incident where you leak information that you weren't supposed to leak because you trained on customer data or my information was somehow exposed to my teammates's information even though if we work in the same team I'm like wait hold on like like [laughter] I am not giving any of this to to a model. Uh so yeah I mean uh I think
64:38enterprises want cheap and perfect and private let's call it right those are the three things and so unfortunately that skews towards the system sites today uh but you know that's not to block like companies like engram and trajectory from doing uh very good PC's with some of the large enterprises I think it's still at the PC stage but you know that even at this level like PC is