Article · 2026-08-10 · 6 moments

5 useful things you'll learn in my new post-training textbook (shipping now!)

After a few long years of finding time to document my lessons from training open models, my post-training book is done! ✦ AI generated

Read original ↗

HOST Nathan Lambert

Claim

Knowing the history of a field better than anyone is what lets a practitioner make the best predictions.

The author argues that knowing the history of one's field better than anyone enables the best predictions, and the book walks readers through three eras of RL on preferences, from its origins in the alignment field to the era exploited by ChatGPT after 2023.

transcript

Nathan Lambert: As you become an expert, knowing the history of your field better than anyone is what lets you make the best predictions (Bill Gurley gives similar advice in his recent book). ... The book will walk you through 3 eras, when researchers learned to do RL on preferences generally until ~2018, spent a few years learning how to apply it to language models from 2019 to 2022, and from 2023 on exploited the examples set by ChatGPT.

explains mechanism · 1

Mechanism

Modern RL systems balance a few core problems — how off-policy the data is, training-inference mismatch, and throughput — using asynchronous RL with separate learner and actor GPUs.

Most of modern RL is a systems problem balancing how off-policy the data is, training-inference mismatch, and throughput, built on asynchronous RL with separate GPUs for learners and actors, a design that has stayed similar for years.

transcript

Nathan Lambert: Most of modern RL is a systems problem balancing a few problems — how off-policy the data is, training-inference mismatch, and throughput. The core systems design, asynchronous RL with separate GPUs for the learners (the GPUs which take gradient steps) and actors (the GPUs which generate the rollouts in the environment), has been similar for a few years.

Fact

My post-training textbook 'Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs' is complete and now shipping via Manning, Amazon US, and (in October) Amazon UK.

The author announces the completion and release of his post-training book, published by Manning, and details its distribution across Manning, Amazon US now, and Amazon UK in October.

transcript

Nathan Lambert: After a few long years of finding time to document my lessons from training open models, my post-training book is done! It's published by Manning, under the title Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs. ... The book is shipping from Manning and Amazon US now, and from Amazon UK in October.

explains mechanism · 2

Claim◆

Post-training-based RL intuition, from the policy-gradient theorem through PPO to modern versions like GSPO and CISPO, is essential for judging whether a new algorithm is fake or has real potential.

Roughly 25% of the book teaches how to think about RL algorithms, from the policy-gradient theorem to PPO and modern variants like GSPO and CISPO, framing this intuition as crucial for telling whether a new algorithm is fake or genuinely promising.

transcript

Nathan Lambert: By word or page count, the book is about 25% RL. This seems appropriate. If there's one thing the book is doing it's teaching people how to think about various RL algorithms. This intuition, from the policy-gradient theorem to PPO to modern versions like GSPO and CISPO, are crucial to understanding if a new algorithm is fake or has potential (no new algorithm will be proven right out of the gates).

extends · 1

Claim

The book communicates the intuitions and history behind post-training, explaining in simple terms why post-training works, the trade-offs to get it right, and the misconceptions people get stuck on.

Most of the book is about communicating intuitions and history of post-training, aimed at explaining why post-training works, the trade-offs needed, and the misconceptions people fall into, with an intentionally higher-voice explanatory style than typical textbooks.

transcript

Nathan Lambert: Otherwise, most of the book is about communicating intuitions and history. Much of the LLM industry is defined by core techniques that haven't changed much in the last few years. This book was my attempt to explain in simple terms why post-training works, what trade-offs people need to make to get it right, and what misconceptions people often get stuck on.

Context

The book existed because critical post-training methods, such as rejection sampling, outcome reward models, and character training, had no foundational online material explaining them and still lack such resources today.

The book began as a website documenting post-training methods that had little to no foundational online material, like rejection sampling, outcome reward models, and character training, which helped make it popular as one of the few places covering these topics intuition-first.

transcript

Nathan Lambert: The book started as a website where I wanted to document key methods of post-training that had potentially no online material explaining them. If there was something, I couldn't find it. This existed for more topics than you would expect, given post-training was already popular in 2024 (when I bought the domain), and continues to this day. Topics like rejection sampling, outcome reward models, and character training are prime examples.

explains mechanism · 1

Highlight slides

RL Intuition as the Book's Core✦ from: Post-training-based RL intuition, from the policy-gradient theorem through PPO to modern versions like GSPO and CISPO, is essential for judging whether a new algorithm is fake or has real potential.Intuition's Purpose✦ from: Post-training-based RL intuition, from the policy-gradient theorem through PPO to modern versions like GSPO and CISPO, is essential for judging whether a new algorithm is fake or has real potential.

Related episodes

Video · The Cognitive Revolution

Alignment with Awakening: Davidad on Moral Realism, AI Wisdom, & why His p(Doom) is Down to 5%

→

Video · The Cognitive Revolution

Exploring the J-Space

→

Video · The Cognitive Revolution

AI:AM Highlights: Exploring the J-Space, AI Superforecasters, SambaNova's Chips, & LTX Video Gen

→