Video · 2026-07-28 · 42m · 6 moments

Solving the Hardest Problem in Robotics | Fei-Fei Li with a16z

✦ AI generated

timeline · colored by role

Definition◆

The next frontier of AI is spatial intelligence — creating AI that can generate, understand, reason with, and interact with physical or virtual spaces.

Fei-Fei Li defines spatial intelligence as AI's next frontier and explains that World Labs' mission is to build large world models toward that goal.

transcript

Fei-Fei Li: World Lab has is a two-year-old startup. I I I think we should just recognize it's a frontier model lab. It's uh we we are building the next frontier of AI which is what we call spatial intelligence. And uh spatial intelligence is about um creating AI that has the ability to generate uh understand reason with and interact with spaces whether it's physical or virtual. And of course uh a means to an end towards spatial intelligence is building large world models. And that's what uh World Labs is mostly focused on.

gives example · 2

Mechanism

Mapping real environments into aligned digital worlds enables replacing all real-world robotics data with scalable digital data, solving the key training and evaluation bottlenecks in general-purpose robotics.

Yinuo explains Scenix's real-to-sim-to-real pipeline: map physical environments into aligned digital worlds where training and evaluation data can be generated at scale, bypassing real-world constraints.

transcript

Yinuo: So for cynics the unique opportunity we see is that there has been a lot of like a bottlenecks. Right now we see faced by the developments of general purpose robots especially around training and also around evaluations. So as we are developing what we call a real to sim to real pipeline. Okay. We want to map the real environments into the digital world that has the best alignments with the real environments. By alignments we mean that whatever happens in the digital world is also going to happen in the real environments such that we can replace all the data all the evaluation we need in the real environment by using the data that can generate at a scalable way in our digital world. So that is how everything started in Synex.

extends · 1gives example · 1

Fact

The lack of data in training and evaluation is the most profound problem in robotics, fundamentally different from language models where data is abundant, and unlocking scaling laws for robotics depends on solving it.

Fei-Fei Li identifies data scarcity as the core obstacle in robotics — unlike LLMs, robotics has no abundant internet-scale dataset, making it impossible to naively apply scaling laws.

transcript

Fei-Fei Li: Really what Synix team is doing is trying to solve this extremely difficult problem in robotics which is the lack of data. M >> the lack of data in training, the lack of data in uh evaluation. This is very very different from language models where data is abundant on the internet. And we know that um in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from? This is something that that is a profound problem that everybody's battling with in in robotics.

explains mechanism · 1extends · 1

Mechanism◆

Simulation plays an essential role in robotics that real-world data cannot replace because it enables counterfactual reasoning — playing out events that haven't happened or cannot happen — and real-world deployments like Waymo already prove its value.

Fei-Fei Li argues there is no binary choice between simulation and real-world data: simulation is essential for counterfactual reasoning, and Waymo's billions of simulation hours prove it works at scale.

transcript

Fei-Fei Li: I want to add to this and be slightly philosophical here is there isn't a a a binary choice between simulation or no simulation. All this come um in together um to to make robotics work. Think about human intelligence. We do a lot of simulation in our head. You know why? There's a very important role simulation plays that real world data doesn't play which is counterfactual reasoning is that you play out events that c hasn't happened or cannot happen or you don't have enough data to make it happen in real world. And while you play it out, you learn how to act in it. Humans do this all the time. We probably don't, you know, we just I know you were at World Cups. I was at the World. Congratulations to Spain winning. I'm sure in the planning of every game there is simulation, whether it's digital or on the on the whiteboard or whatever, that simulation, the role simulation plays is counterfactual reasoning. And that's really important in robotics because we just do not have cannot possibly have enough real world data for that. Here's a real life uh example the the industry of self-driving cars. Whimo has officially said they use billions of hours of simulation and and actually Whimo is more simulationheavy than just real world data heavy. So these are real examples and and as you know Martin Andrew too cars are the simplest kind of robots. Yeah. So so clearly simulation plays a huge role in robotic learning.

Context

Scenix is building infrastructure — not robots — that is naturally model-agnostic and embodiment-agnostic, providing the digital worlds where any robot type can learn and evaluate.

Yinuo clarifies Scenix does not build robots or robot brains — it builds model-agnostic, embodiment-agnostic digital infrastructure so any company can place their robot into a simulated world for training and evaluation.

transcript

Yinuo: So what we have been building you can imagine is a infrastructure like with the softwares around this infrastructures for people to for them build words such that robot can learn and evaluate and this infrastructures is naturally model agnostic and embodiment agnostic. I just want to be very clear just because this is actually a very subtle for you it's obvious but it's a very subtle point which is um from what you said that's not building a robot it's building an environment which another company can place their robot brain to navigate and to learn.

Prediction

Robotics deployment in the real world follows a clear progression from fully structured environments like factories, to semi-structured environments like warehouses, to unstructured environments like homes — and the most pragmatic approach is to focus on semi-structured environments first.

Yinuo describes the natural deployment trajectory of robotics — structured to semi-structured to unstructured — arguing the pragmatic path is to solve semi-structured settings like warehouses before tackling homes.

transcript

Yinuo: So that's a very good question. So if you look at for example all the uh progressions of robotic applications in the real environments it has always followed the trend from going from fully structured environments into semiructured environments and then into unstructured environments. For fully structured environments what do we mean that you have knowledge and control over all the configurations within the environments like factories like factories or for car manufacturing mice those has been automated for decades. Yeah. Yeah. Yeah. And then you have for example semiructured environments which you have certain controls over the environments for example like the Amazon warehouses or for example like restaurants hotels where you have certain control over the environment to just make the task easier for your robots but there are obviously many other like objects or for example clothes those are the object you don't have control and then for the unstructured environments it's like your home and my host those is I would say the grand challenge. If you're thinking about where does the robustness coming from, robustness coming from a sufficient coverage of the scenarios that robots might encounter. So, it's so much easier and more approachable at least like right now to focus more on the semiructured environments before we move on to fully unstructured environments. So, we will move into that direction. It's just we want to uh take a more sustainable and more realistic approach towards it.

supports · 1

Highlight slides

Spatial Intelligence: AI's Next Frontier✦ from: The next frontier of AI is spatial intelligence — creating AI that can generate, understand, reason with, and interact with physical or virtual spaces.Large World Models as the Path Forward✦ from: The next frontier of AI is spatial intelligence — creating AI that can generate, understand, reason with, and interact with physical or virtual spaces.No Binary Choice✦ from: Simulation plays an essential role in robotics that real-world data cannot replace because it enables counterfactual reasoning — playing out events that haven't happened or cannot happen — and real-world deployments like Waymo already prove its value.Simulation Enables Counterfactual Reasoning✦ from: Simulation plays an essential role in robotics that real-world data cannot replace because it enables counterfactual reasoning — playing out events that haven't happened or cannot happen — and real-world deployments like Waymo already prove its value.Waymo Proves It at Scale✦ from: Simulation plays an essential role in robotics that real-world data cannot replace because it enables counterfactual reasoning — playing out events that haven't happened or cannot happen — and real-world deployments like Waymo already prove its value.

Related episodes

Video · 20VC

The Best AI Companies Have Unique Data Acquisition Strategies | Simile Co-founder & CEO

→

Video · The Cognitive Revolution

LTX CEO on Video Gen + Can we beat AI superforecasters?

→

Video · 20VC

The AI Company Simulating the Entire Economy | Simile Co-founder & CEO, Joon Sung Park

→