RLVR trains language models by having them attempt verifiable tasks like math or code, using measured accuracy on those tasks as the reward signal for reinforcement learning updates.
Nathan lays out the core mechanism of Reinforcement Learning with Verifiable Rewards: the model generates completions to problems with checkable answers, and correctness becomes the reward that drives training. ✦ AI generated
Nathan Lambert · Lex Fridman · 2026-01-31 · original ↗
starts at this moment · 99:18
“Can you describe RLVR, popularized by DeepSeek R1? Can you describe how it works?”
Reinforcement learning is classically an agent that acts in an environment, and the environment gives it a state and a reward back, and you try to maximize this reward. In the case of language models, the reward is normally accuracy on a set of verifiable tasks, whether it's math problems or coding tasks.
verbatim transcript · starts at 99:18
99:18classically an agent that acts in an environment, and the environment gives it a state and a reward back, and you try to maximize this reward. In the case of language models, the reward is normally accuracy on a set of verifiable tasks, whether it's math problems or coding tasks. And it starts to get blurry with things like factual domains. That is also, in some ways, verifiable, or constraints on your instruction, like respond only with
99:45words that start with A." All of these things are verifiable in some way, and the core idea of this is you find a lot more of these problems that are verifiable and you let the model try it many times while taking these RL steps, these RL gradient updates. The infrastructure evolved from reinforcement learning from human feedback, where in that era the score they were trying to optimize was a learned reward model of human
100:12preferences. So you kind of changed the problem domains and that let the optimization go on to much bigger scales, which kind of kickstarted a major change in what the models can do and how people use them. - What kind of domains is RLVR amenable to? - Math and code are the famous ones, and then there's a lot of work on what is called the rubrics, which is related to a word people might
100:36have heard: LLM-as-a-judge. For each problem, I'll have a set of problems in my dataset. I will then have an LLM and ask it, "What would a good answer to this problem look like?" And then you could try the problem over and over again and assign a score based on this rubric. That's not necessarily verifiable like math and code domains, but this rubrics idea and other scientific problems that might be a little bit more vague is where
101:03the attention is, where they're trying to push this set of methods into these kind of more open-ended domains so the models can learn a lot more. - I think that's called reinforcement learning with AI feedback, right? - That's the older term for it coined in Anthropic's Constitutional AI paper. It's like a lot of these things come in cycles. - Also, just one step back for RLVR. I think the