How NVIDIA Builds Open Models for the Age of AI

Bryan Catanzaro: A hybrid architecture swaps most of the attention layers in the Transformer with Mamba layers and keeps a few blocks with full attention. NVIDIA's models rely on the same hybrid pattern. Most of the layers are Mamba, which keeps long inputs cheap to process and is what makes a million-token context window practical rather than just theoretical. A few attention layers are placed in between to restore the pinpoint recall that Mamba gives up, so the model can still retrieve exact details from anywhere in the context. On top of the hybrid design, the models also use mixture-of-experts, or MoE, layers. An MoE layer consists of many small expert layers and routes each token to only a few of them. So instead of running every parameter on every token, the model activates only a small subset of its parameters. That allows us to use a large total capacity at a low per-token cost, which keeps the model fast.

Bryan Catanzaro: Its larger models are pretrained in a 4-bit number format called NVFP4, meaning most of the math during training uses just four bits per value. Fewer bits means less memory and less data to move, so the math runs faster and on less power. Note that four bits is very little precision. Most teams train in higher precision and only shrink the model afterward, which loses accuracy. NVIDIA trained in 4 bits from the first step because it knew its next GPU generation, Blackwell, was being built with fast 4-bit hardware. The model and the chip were designed for each other. The first step to inventing something amazing is believing that you can. With those efficiency choices in place, post-training is where the model gains its capability. Scaling up this reinforcement learning stage has been shown to improve a model's capability significantly. Because its models are cheaper to run, it can scale RL at a lower cost, letting the model practice across many environments in parallel and learn from over a million rollouts. This is where the VP placed the real bottleneck for better models. Not data or compute. The diversity of these training environments.

Bryan Catanzaro: You probably know NVIDIA sells GPUs. But did you know it is also the largest publisher of open AI models in the world? Its models rank among the most downloaded on Hugging Face, and the lineup goes way beyond chatbots: reasoning models, world models, models for humanoid robots, self-driving cars, even drug discovery, quantum computing, and global forecasting.

Bryan Catanzaro: Nemotron is not just a model. It is also the data, the tools, the training recipes, and the ideas published in the papers. The finished weights are just the last step. In practice that means NVIDIA publishes the training datasets, the post-training datasets, the reinforcement learning environments, and the recipes needed to reproduce the work. A team can not only run the model, but also see how it was built and train their own version. Releasing data at this scale is unusual. It is a scary thing for a company to do. NVIDIA does it anyway, and even shares clever building blocks like synthetic 'persona' data that is statistically realistic but fully private, so a model can learn about the world without memorizing real people. You can see this in what the community has done with it. NVIDIA's open data and models have spawned a large set of community fine-tunes and derivatives. One of its robotics datasets became one of the most-downloaded datasets on Hugging Face. Many variants of its Nemotron models have millions of downloads each, and the Nemotron family recently crossed 100M in total. You cannot build like that from the weights alone.