The Next Frontier of AI Is Spatial Intelligence | Fei-Fei Li on a16z

Yunzhu Li: As we are developing what we call a real to sim to real pipeline. We want to map the real environments into the digital world that has the best alignments with the real environments. By alignments we mean that whatever happens in the digital world is also going to happen in the real environments such that we can replace all the data, all the evaluation we need in the real environment by using the data that can generate at a scalable way in our digital world.

Fei-Fei Li: There isn't a binary choice between simulation or no simulation. All this come together to make robotics work. Think about human intelligence. We do a lot of simulation in our head. Why? There's a very important role simulation plays that real world data doesn't play which is counterfactual reasoning — you play out events that hasn't happened or cannot happen or you don't have enough data to make it happen in real world. And while you play it out, you learn how to act in it. Humans do this all the time. Here's a real life example: the industry of self-driving cars. Waymo has officially said they use billions of hours of simulation and actually Waymo is more simulation-heavy than just real world data heavy. So clearly simulation plays a huge role in robotic learning.

Yunzhu Li: If you look at all the progressions of robotic applications in the real environments it has always followed the trend from going from fully structured environments into semi-structured environments and then into unstructured environments. Fully structured environments means you have knowledge and control over all the configurations within the environments like factories. Then semi-structured environments where you have certain control over the environment like Amazon warehouses or restaurants or hotels. Unstructured environments is like your home. That is the grand challenge. Robustness comes from a sufficient coverage of the scenarios that robots might encounter. It's so much easier and more approachable at least right now to focus more on the semi-structured environments before we move on to fully unstructured environments.

Yunzhu Li: I think it's going to take a very long time. If you really think about robots in the real environment, it will always be a system. Every working robot in the real environment is a system. You need to be very mindful and thoughtful about how the system comes together: the hardware, the software, the brain, even the details of the friction coefficients of your fingers. There's a lot of things you have to consider to make these things a reality. The state-of-the-art is always moving faster than I expected, so what I'm focusing on is very different from when I started my PhD. But we also have to be calibrated about our predictions. To achieve human level efficiency and capabilities, it will take longer.