Preference Data: The Most Opaque Part of Post-Training | RLHF & Post-training Course, Lecture 8
This time we dig into the philosophical and practical underbelly of post-training. Discussing how economics, philosophy, and control theory ideas converged into modern RLHF techniques. We also discuss opaque details of modern human data collection. This one is a fun one for me, because I get to link to a lot of papers I really enjoyed writing a few years ago. Welcome to The RLHF Book & Post-Training Course with Nathan Lambert. Ask questions and I'll answer them in the next roundup video! Slides for this lecture are here: https://rlhfbook.com/teach/course/lec... Chapters: 00:00 Intro & context 07:34 A short history of preferences 20:17 A brief overview of preference data 31:11 Open questions in RLHF data Some papers and resources referenced: The History and Risks of RLHF (Entangled Preferences) — Lambert et al., 2023: https://arxiv.org/abs/2310.13595 Objective Mismatch in Model-based RL — Lambert et al., 2020: https://arxiv.org/abs/2002.04523 The Alignment Ceiling: Objective Mismatch in RLHF — Lambert & Calandra, 2023: https://arxiv.org/abs/2311.00168 InstructGPT labeler instructions (PDF): https://rlhfbook.com/assets/instructg... All resources will be available at https://rlhfbook.com/ Order a copy of the book (physical recommended) on Manning.com: https://hubs.la/Q03Tc3dc0 Order a copy on Amazon: https://amzn.to/4cwCDJQ With specific course resources at https://rlhfbook.com/course (recording links, slides in PDF and native form, etc.) And code at https://rlhfbook.com/code Get more information on Nathan at http://natolambert.com/ and stay up to date with his work on Interconnects https://www.interconnects.ai/ Course YouTube playlist: • Welcome to The RLHF Book & Post-Training C... Join the book's Discord Community: / discord Nathan is on… X: / natolambert LinkedIn: / natolambert GitHub: https://github.com/natolambert BlueSky: https://bsky.app/profile/natolambert.... Threads: https://www.threads.com/@natolambert Substack: https://substack.com/@natolambert Slides are built with Colloquium: https://github.com/natolambert/colloq... Thank you to my many collaborators who helped me learn this information I get to share with the world!
RLHF and Post-training Overview | RLHF & Post-Training Book Course, Lecture 1
On-Policy Distillation & Using Synthetic Data in Post-Training | RLHF Book Course, Lecture 7
Direct Preference Optimization (DPO) and Friends | RLHF & Post-training Course, Lecture 6
I Built an LLM From Scratch
Obsidian Note-Taking System for Academics & Students - Full Tutorial & Demo (Write Papers Faster)
Is Fine-Tuning Still Needed? LLMs, RAG, & LoRA
Why Netflix is betting on systems thinkers—not specialists—in the AI era | Elizabeth Stone (CPTO)
Unfortunately, You Need to Know What the Jevons Paradox is
CHOSEN ONE!! YOUR IDENTITY REVEAL JUST SHOOK THE INTERNET... AND THEIR MINDS
The Rise of Reasoning Models | RLHF & Post-training Course Lecture 5
GRPO's new variants and implementation secrets
Understanding Policy Gradient Algorithms for RL on LLMs | RLHF & Post-training Course Lecture 3
Keynote: After the AI Hype – What’s Real, and What’s Next - Richard Campbell - 2026
ML Foundations (prerequisites) for Post-Training | RLHF Book Course, Lecture 0
Musica para trabajar activo y alegre - Música Alegre para en Tiendas, Cafés | Deep House Mix 2026
How To Think SO Clearly People Assume You're Brilliant
Last Lecture Series: “How to Win Without Crushing Your Soul” - Graham Weaver
How Small Models Learn to Think Like Giants
Reinventing Entropy | Compression is Intelligence Part 1
Yann LeCun: World Models: Enabling the next AI revolution
This Harvard Textbook Reveals How the Top 1% ACTUALLY Build Their Minds
