AlphaFold 2 succeeded by rejecting the 'bitter lesson' and instead building in extensive domain-specific, hand-engineered structure, because unlike language models, protein data is finite.
Jumper argues AlphaFold 2 is a counterexample to the 'bitter lesson' — success came from deliberately encoding biological and geometric domain knowledge into the architecture, because protein data (unlike text) is finite. ✦ AI generated
John Jumper · Machine Learning Street Talk · 2026-06-22 · original ↗
starts at this moment · 39:10
I don't really love the bitter lesson as people try and apply it. In fact, AlphaFold 2 is the opposite of that. We did a whole bunch of specialty stuff because our data is not finite. And in fact, now that we've gone to language models, we found our data is still finite. The internet is finite.
verbatim transcript · starts at 39:10
39:10an answer for no matter how hard this problem is. And maybe we should give it some more layers. And my GPU's out of memory, so maybe I should just, you know, run it back through so I don't have to um have more memory. But even without that, I think AlphaFold even without recycling was learning this kind of iteration. And then we put in a kind of code idea, architectural idea to
39:35help this process that it was going to learn from the data. Um going back to the earlier thing about exactly how far residues are apart, we didn't tell AlphaFold that. We knew that the data would scream at it that I and I plus one were 1.3 angstroms apart. So, I think when we think about our human understanding, I think one of the you know, I don't really love the bitter lesson as
40:01people try and apply it. In fact, AlphaFold 2 is the opposite of that. We did a whole bunch of specialty stuff because our data is not finite. And in fact, now that we've gone to language models, we found our data is still finite. The internet is finite. So, I think, you know, don't do architectural research is the wrong thing to draw from it, but have some humility about which
40:21things go into your code and which things will be derived from your data. Look at what's missing, understand the algorithm that deep learning that the deep learning is trying to learn. How can you accelerate it? How can you add hypotheses and where you specially you add kind of communication. The most important thing we would do within the architecture is modify which units communicated and how. I think all of
40:43these have been kind of how we derive understanding to ultimately make an iterative process. And it should shock no one that if you're trying to make an intricate geometric object that you are going to iterate. Or similarly, if you think about um generating text, right? And one kind of naive assumption that people will make is that these are next word generators, so they have no idea what's going to happen in two words
- ·Jumper argues AlphaFold 2 is the opposite of the bitter lesson
- ·Success came from extensive domain-specific, hand-engineered structure
- ·Protein data, unlike text, is finite — requiring specialist architecture
- ·Biological and geometric domain knowledge was deliberately encoded
- ·Bitter lesson (scaling general methods) would have failed here
- ·Finite data demands architectural priors, not brute-force scale