Balsam highlights what he calls his favorite result from the predictive data debugging paper: that data filtering and activation-based reward shaping are two sides of the same coin, producing approximately equal effects on off-target learning, giving practitioners two roughly equivalent options. ✦ AI generated
Dan Balsam · The Cognitive Revolution · 2026-08-08 · original ↗
starts at this moment · 8:17
I think like my favorite thing from that paper was that like the researchers involved in that paper like show that there's like sort of a pretty deep isomorphism between those two things. Like they they kind of are two sides of the same mountain like filtering the data and reward shaping and they achieve like approximately the same effects and approximately the same amount of offtarget effects as each other.
verbatim transcript · starts at 8:17
8:15if it was going to be trained on it. And so looking at the features and understanding that those features like correlate with downstream behaviors, if there are features that are surprisingly upweighted by some data, then it can tell you what it can be like fairly predictive about whether the model is going to learn some offtarget effect from that data. And so I think one of the most interesting things from that
8:35paper is like we explored a bunch of different mitigation methods. So we looked at reward shaping which is training or involving some reward process that comes from the activations of the model itself. So like hey learn what you're going to learn from this data but maybe a penalty if you're learning this particular feature. And we also looked at data filtering. And I think like my favorite thing from that
8:57paper was that like the researchers involved in that paper like show that there's like sort of a pretty deep isomorphism between those two things. Like they they kind of are two sides of the same mountain like filtering the data and reward shaping and they achieve like approximately the same effects and approximately the same amount of offtarget effects as each other. So like your option your options if you like
9:23have a behavior and you do some like predictive data debugging and you think the model's going to learn and you don't want it to learn or its behavior is going to change in a bad way your options could be filter your data if you have enough data or it could be like intervene in the training process in some way and yeah I think this is something that we've expanded on since
9:40then I think it's pretty exciting to think about it applications to like RL for instance and in RL like we did it with DPO but like true RL like the difference is like some rollouts may contain information that you don't want the model to alert, you know, um even subtly, you know, >> I don't know what you're are you referring to something. >> Yes. Yeah. It's topical. And so being