Waymo's structured representations enable runtime validation, efficient simulation, and verifiable training feedback — benefits that pure end-to-end approaches cannot match — while Tesla's learned representations capture nuance that no fixed schema can anticipate.

How each system converts raw sensor data into an internal world model is one of the most consequential architectural decisions: Waymo keeps an explicit, inspectable list of objects and road elements, while Tesla derives everything through neural networks producing 1,000 distinct tensors per time step. ✦ AI generated

Article Author · ByteByteGo Newsletter · 2026-08-17 · original ↗

Something has to convert millions of pixels and points into a description that the system can understand and operate on. The nature and form of that description is one of the most important architectural decisions for an autonomous driving setup. The system maintains compact structured representations, meaning explicit lists of objects, their semantic attributes, and roadgraph elements describing lanes and connections. The Waymo engineering team provides three reasons for such a setup: Correctness and safety validation can run at inference time, while the vehicle is moving; Simulation runs efficiently at large scale, because a compact world state is cheap to replay and modify; Training feedback becomes verifiable, since a component evaluating driving quality has something concrete to measure. Tesla's documentation talks about per-camera networks performing semantic segmentation, which assigns every pixel to a category, plus object detection and monocular depth estimation, meaning distance estimated from a single camera. Those feed birds-eye-view networks that output road layout, static infrastructure, and three-dimensional objects in a top-down view. A full build involves 48 networks taking nearly 70,000 GPU hours to train and producing 1,000 distinct tensors per time step. The trade-off is that a structured representation can be inspected, logged, replayed, and checked against explicit criteria. But it limits what the system can express. On the other hand, a learned representation carries nuance no schema can anticipate.

Read full article ↗excerpt · fair-use quotation