Fei-Fei Li is Solving the Hardest Problem in Robotics | World Labs with a16z

Yunzhu Li: What Synix team is doing is trying to solve this extremely difficult problem in robotics which is the lack of data — the lack of data in training, the lack of data in evaluation. This is very very different from language models where data is abundant on the internet. And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from? This is a profound problem that everybody's battling with in robotics. As Cynics, we are developing what we call a real to sim to real pipeline. We want to map the real environments into the digital world that has the best alignments with the real environments. By alignments we mean that whatever happens in the digital world is also going to happen in the real environments, such that we can replace all the data, all the evaluation we need in the real environment by using the data that we can generate at a scalable way in our digital world.

Yunzhu Li: If you put things more specific, simulation can provide two levels of benefits. The first one is reliability and the second one is efficiency. For reliability, if you're thinking about a robotic system working reliably in the real environments, you need data to provide systematic coverage of all the state space and the variations that robots might encounter. That's how you can learn to be robust. So with simulation, you can do systematic randomizations and control over the variations of lighting, frictions, geometries, object types and also all different kind of physical parameters to make sure you have sufficient coverage of the state space. Second is about efficiency. Right now many people are doing teleoperation — you're collecting the data at a speed that is actually slower than humans actually doing the task. But for many of our clients, human speed is not good enough. They want faster than human speeds. For the robot to move faster, it's not as simple as just drive the robot faster because the gravity doesn't change. But in simulation, you can do systematic speed up of the robots behaviors to train the robots such that it considers all the dynamics changes of the environments.

Yunzhu Li: What we have been building you can imagine is infrastructure with the software around this infrastructure for people to build worlds such that robots can learn and evaluate, and this infrastructure is naturally model agnostic and embodiment agnostic. For our customers right now they have all different kinds of robots — some are using a single robot arm, some are using a bi-manual, some are using a fixed arm, some are using mobile manipulators, some using grippers. Our platform is naturally embodiment agnostic. We can very easily integrate different kinds of robotic embodiments, put them into the worlds we generated, and be able to give those individual robots capabilities of doing the right tasks at the right levels of reliability and efficiency in the real environments. We are also model agnostic — we can use the data generated by our worlds to train different models either from scratch or doing post-training of existing foundation models like vision language action models or world action models. To us it doesn't matter, we just want to make sure we have the infrastructure and the worlds such that the robot can work reliably in the real environment.

Yunzhu Li: If you look at all the progressions of robotic applications in the real environments, it has always followed the trend from going from fully structured environments into semi-structured environments and then into unstructured environments. For fully structured environments we mean you have knowledge and control over all the configurations within the environments — like factories or car manufacturing — those have been automated for decades. And then you have semi-structured environments where you have certain controls over the environments — for example Amazon warehouses or restaurants, hotels — where you have certain control to just make the task easier for your robots but there are many other objects you don't have control over. And then for the unstructured environments it's like your home and my home. That is the grand challenge. If you're thinking about where does robustness come from, robustness comes from a sufficient coverage of the scenarios that robots might encounter. So it's so much easier and more approachable at least right now to focus more on the semi-structured environments before we move on to fully unstructured environments.