The more RL we throw at models, the more we should expect reward seeking to increase, and there is economic pressure toward ontological drift — models internalize concepts in ways that become increasingly illegible to humans due to length penalties and optimization for efficiency.
Alex explains that as RL increases, models face pressure to cram meaning into fewer tokens due to length penalties, driving ontological drift away from human-interpretable representations. Models already show increasingly weird and hard-to-interpret language in their chain of thought. ✦ AI generated
Alex · Machine Learning Street Talk · 2026-07-31 · original ↗
starts at this moment · 28:32
There is also a lot of pressure uh on the models to internalize concepts very very strongly. There is uh length penalty that you have to have during RL uh otherwise you know the chain of thought would just get longer and longer. Um and uh when we used to be in the pre-training compute dominated era, it was sort of not that important to penalize the the COD. But the more inference costs are important and the more uh post-training compute gets applied, the more economic pressure there is to crank the um length penalty as high as you possibly can. So on in terms of optimization pressure on the model that results in cutting everything that's superfluous and uh in the limit. So if you if you imagine this this length penalty went to infinity you should expect something like maximal entropy across all tokens so that you try to cram as much meaning as you can into the tokens. Um to give an example, we we in in our previous work um that we released on um last last fall on this anti- scheming project where we try to like train a model not to be deceptive and then kind of like see what happens. We looked at lots of like transcripts and also in in in this project and what we do see is that like the language models um do start having like this kind of weird language that becomes increasingly hard to interpret and this is just the verbalized reasoning.
verbatim transcript · starts at 28:32
28:32pre-training compute dominated era, it was sort of not that important to penalize the the COD. But the more inference costs are important and the more uh post-training compute gets applied, the more economic pressure there is to crank the um length penalty as high as you possibly can. So on in terms of optimization pressure on the model that results in cutting everything that's superfluous and uh in the limit.
28:59So if you if you imagine this this length penalty went to infinity you should expect something like maximal entropy across all tokens so that you try to cram as much meaning as you can into the tokens. Now it's not clear to what extent this is happening but uh this is a direct pressure that we should just expect to continue going up. Yes. And it is conceivable that this process
29:23is accelerating over time especially like everyone is now talking about continual learning you know because right now at least we have a centralized model right so so we have we have red teaming we have you know frontier companies building these these models testing them it's going to become far more diffused and decentralized and all of this kind of stuff I mean what what is your prescription and what is your
29:39prognosis >> yeah I think um well one thing is just dedicating more resources into like investigating this phenomenon getting better ways of measuring like the first thing when you want to make an in intervention on something is having a really good measurement of it. And that's essentially the vision we had for the project um of trying to establish that. But we need far more work making it uh more robust and applying it to uh
30:03even more frontier models and so on. Um and then second like just making it a default thing that the labs track. They all like track how reward seeking are and different various measures. Anytime the model in training faces a trade-off between doing what it believes is intended versus doing what it believes is rewarded, it gets by definition rewarded for the cases where it ignores the actual intent. So the more RL we