Beck explains that energy-based models differ from standard neural nets because their cost function also constrains internal/latent states, not just the input-output mapping, requiring a dual minimization process. ✦ AI generated
Jeff Beck · Machine Learning Street Talk · 2026-01-25 · original ↗
starts at this moment · 13:58
“Jeeoff, let's talk about energy based models.”
So in a traditional neural network, you take in your inputs, you got your outputs, and the cost function is just a function of the inputs and the outputs. And the only thing that you're optimizing is the weights. In an energy based model, there's another thing that your cost function operates on, and that's one of the internal states of your model.
verbatim transcript · starts at 13:58
13:58cost function is applied. Right? So in a in a traditional neural network, you take in your inputs, you got your outputs, and the cost function is just a function of the inputs and the outputs. And the only thing that you're optimizing is the weights. In [snorts] an energy based model, there's another thing that that your cost function operates on, and that's something one of the internal states of your model. And
14:19as a result like in order to figure out what the best you know the the the best approach is right you actually have to do two minimizations. One that that finds the energetic minimum associated with the the the part of the cost function that operates on the internal states like the hidden nodes of your network right and then one that is the prediction that is your like effective
14:37prediction error. Um this is this is very much consistent with the approach that a basian would take right you have a you have a a prior probability distribution which gives you an energy function over every single latent variable in your model and you are optimizing with respect to all of them. So so you take a probabistic approach good examples of this are like a variational autoenccoder. A variational
14:56autoenccoder I think is a is the best example of the most commonly used energybased model out there. Why? because you have an encoder network, you have a decoder network, right? And your cost function is based on the difference between inputs and outputs, right? So that's just like a that's fine. That's still a regular, but it also is how how Gaussian in a well, it depends on what
15:15flavor of V8, but you also have some uh some some part of your cost function um is a function of the actual rep internal representation, right? In a traditional VAE, it's it's how Gaussian is. You want that internal representation to be as Gaussian as possible. Um if it's a VQ VAE then it's like mixture of Gaussians but it's still like a cost function that is applied on the internal states as
15:37well as on the inputs and outputs. >> Very cool. So a VAE is is a fairly cononical example of an energy based model and what you were saying about the I mean you know the whole DL world is obsessed with test time inference at the moment and in a way that that is a step towards what you're talking about. So yeah, you're treating a certain Yeah, you're treating some of the weights of
15:54your model, right? I mean, well, yeah, you're treating some of the weights of your model as if they're latent variables, right? Because when you when you show a new input, right, you're allowed to change some of the weights without looking at the output, right? And so what are you doing? Well, you're treating the weights as latent. Now, I think that like which makes it a great trick in my opinion. It's like, oh,