Training Robots in Worlds That Don't Exist | World Labs with a16z

Yunu Guo: They don't contradict with each other. Simulation is essentially trying to predict how the environment is going to change when you apply the actions—this is a model of the world. It doesn't necessarily have to be pure physics. It can be a combination of both physics and learning. We are collecting real world data. We will be using those real world data. It just at different stages of this data flywheel. At the very beginning we have stronger emphasis on physics to make sure we have the right consistency and right structure for us to train the robot policies. But as we accumulate more and more data both through data collection and through the collaboration with our clients, we'll have the data that will be moving towards more learning-based modeling of the environments. This kind of transition and data flow is really an enabling factor of getting the best of both physics and geometry and consistency as well as all the power and magic from data and compute.

Yunu Guo: There are two levels of benefits simulation can provide. The first one is reliability and the second one is efficiency. For reliability, if you're thinking about a robotic system working reliable in the real environments, you need data to provide systematic coverage of all the state space and the variations that robots might encounter. With simulation, you can do systematic randomizations and control of lighting, frictions, geometries, object types and also all different kind of physical parameters to making sure you have sufficient coverage of the state space. Second is about efficiency. Right now many people are doing teleoperation and you're actually collecting the data at a speed that is slower than human actually doing the task. For many of our clients human speed is not good enough. They want faster than human speeds. For the robot to move faster, it's not as simple as just drive the robot faster because the gravity doesn't change. But in simulation, you can do systematic speed up of the robots behaviors to train the robots such that it considers all the dynamics changes of the environments.

Yunu Guo: If you look at all the progressions of robotic applications in the real environments it has always followed the trend from going from fully structured environments into semi-structured environments and then into unstructured environments. For fully structured environments you have knowledge and control over all the configurations within the environments—like factories, for car manufacturing, those has been automated for decades. Then you have semi-structured environments where you have certain control over the environments, for example like the Amazon warehouses or restaurants or hotels. For the unstructured environments it's like your home and my home—that is the grand challenge. If you're thinking about where does the robustness come from, robustness comes from a sufficient coverage of the scenarios that robots might encounter. So it's so much easier and more approachable right now to focus more on the semi-structured environments before we move on to fully unstructured environments.

Yunu Guo & Fei-Fei Li: I think it's going to take a very long time. If you really think about robots in the real environment, in the end it will always be a system. Every working robot in the real environment is a system. You need to be very mindful about how the system comes together—the hardware, the software, the brain, even the friction coefficients of your fingers. There's a lot of things you have to consider to make these things a reality and it will take iterations. Even LLM does not have human brain efficiency. Human brain operates on 30 watts. We are far from that. I don't think we're anywhere close when it comes to robotics.