Foundation models like AlphaFold produce confidently biased predictions on novel questions at the edge of knowledge because training data there is sparse, so they must be paired with statistical methods (like prediction-powered inference) that merge in ground-truth data to yield trustworthy uncertainty estimates.
Using AlphaFold as a case study, Jordan shows foundation models are systematically biased on edge-of-knowledge questions and argues ground-truth data must be statistically merged in to correct this. ✦ AI generated
Michael I. Jordan · Machine Learning Street Talk · 2026-05-20 · original ↗
starts at this moment · 21:04
“So I interviewed John Jumper last week at Google and you did some analysis on those 200 million predicted proteins and and and you found they were very good but there was something missing but you could robustify them.”
There needs to be around any foundation model the ability to maybe collect a bit of ground truth data to merge it in with some procedure like this and then to give out a more trustable answer. That's all not science fiction. that's what can be done and what really needs to be done and I'm sure the AlphaFold people are on board with that.
verbatim transcript · starts at 21:04
20:45but it's using this rather highly biased architecture. And it's now it's not biased overall. In fact, its accuracy is high overall. But for the question I'm asking, it might be very biased. And that's going to happen a lot in science because scientists are rarely interested in just studying the past over again. They're interested in brand new things on the edge of knowledge. And that's where specifically these foundation
21:04models will be most poor and most highly biased. So there needs to be around any foundation model the ability to maybe collect a bit of ground truth data to merge it in with some procedure like this and then to give out a more trustable answer. That's all not science fiction. that's what can be done and what really needs to be done and I'm sure the AlphaFold people are on board
21:26with that that they would not find that weird or surprising. Um, but a lot of other people out there talk about bias and all that and they either don't worry about it. I say it'll go away if we have enough data or they just critique the architectures and critique the outputs but they have no scientific I you know method in in mind that'll help us go forward. So that's kind of the state
21:47we're in. I challenged John a little bit about the extent to which AlphaFold understands and he was basically allergic to the word understands. >> We are not trying to tell you everything. We are not a model of the entire cell. These machines let us predict. They let us control. We have to derive our own understanding at this moment. Right? We can experiment now on the artifact. We can look at the
22:15200 million predicted structures. not just the 200,000 experimental structures in order to help us understand, but it doesn't do the act of understanding for us. It does the act of predict and maybe control. >> Why why should AlphaFold understand? >> Well, what would it mean to I mean, he was he was sketching it out to me. He kind of said that this this is a weird alien artifact and it's not like it's
22:35kind of created. It's refined. There's this recycle pathway. You can put the thing through multiple times. you can kind of corrupt it halfway through and the network is just iteratively kind of you know it solves the complex bit first and then it's refining refining refining and like could we interpret that as an understanding process >> I don't think we need to see I think this anthropomorphizing of intelligence
22:55and understanding all that is not necessary not appropriate and is is a distraction for many many problems why say it understands you know some of my heritage comes from seeing in in real life in in industrial settings machine learning algorithms being rolled out 20 30 years ago so When I first went to the west coast, I visited Amazon in around 2000. They were using huge amounts of