Models do not acquire most of their knowledge or capabilities during post-training; post-training (including RL) mostly makes low-likelihood events learned in pre-training more likely, so it only produces small nudges to the weights, and effective intervention must target what data is reinforcing.

6:24underlies a lot of our work is there's a good amount of evidence that models don't most of what a model knows like most of its sort of like knowledge and capabilities come from pre-training what happens in post-training including like RL is mostly making low likelihood events from pre-training more likely and so there's not that much new knowledge or like sort of new capabilities that get baked into models in post-training

6:50process. This is somewhat debated, but I think this is like a view that we think is mostly true and informs a lot of how we think about things. And so, because there's only like sort of relatively small nudges in the weights that are happening in any type of post-raining process, most of the sort of like raw capability to produce whatever outcome postrain is going to do already exists

7:15in the model in some way. I think now we're in an era where things are RL so heavily it's not true but back in the day of like RHF like style post training I think it was pretty well known that base models were sometimes more capable than their like instruction tuned RHF counterparts where there was like a little bit of mode collapse that was happening and so like one way to think

7:36about what's happening in RL training is that like you have a little bit of mode collapse in the instruction tuning phase and you have these base models and they're extremely capable but they're like very weird very hard to prompt So you like make them into a format and that has like a more intuitive humanf facing API, but then you want to bring back out some of those capabilities or

7:55like reinforce some of the capabilities that the model actually learned during pre-training. It's 100% true, but I think it's like probably directionally true. And so with predictive data debugging, I think like basically the idea is like you can look at what a model is already like thinking as it's looking at some data. And that's like relatively predictive of like what that data is going to reinforce in that model

8:15if it was going to be trained on it. And so looking at the features and understanding that those features like correlate with downstream behaviors, if there are features that are surprisingly upweighted by some data, then it can tell you what it can be like fairly predictive about whether the model is going to learn some offtarget effect from that data. And so I think one of the most interesting things from that

8:35paper is like we explored a bunch of different mitigation methods. So we looked at reward shaping which is training or involving some reward process that comes from the activations of the model itself. So like hey learn what you're going to learn from this data but maybe a penalty if you're learning this particular feature. And we also looked at data filtering. And I think like my favorite thing from that