Stanford CS329A (Ep 9): Post-Training Evolution — From Next-Token Chatbots to Autonomous Agentic Fleets (SFT, RLHF, RLVR, and Agent RL)
How did raw, web-scale foundation models — essentially stochastic next-token predictors incapable of carrying on a coherent human conversation — transform into polite conversational assistants, and now into self-directed, tool-wielding autonomous agents capable of modifying production codebases and navigating real-world cloud infrastructures?
In this grand finale of Part 2: Training & Evolution (Lectures 6 through 9) of Stanford CS329A: Self-Improving AI Agents, we synthesize the macroscopic history of modern artificial intelligence through the lens of post-training engineering. Drawing on foundational perspectives from Google DeepMind senior research scientist Melvin Johnson, we examine how the locus of the feedback signal shifted over five evolutionary epochs:
"The history of post-training is the progressive delegation of supervisory authority — moving from human demonstration hands, to human preference eyes, to deterministic rule verifiers, and ultimately to raw physical execution environments."
This post presents the comprehensive Autonomous Agent Evolution Roadmap, dissecting Post-Training LLM Evolution, the transition from Supervised Fine-Tuning SFT Agent baselines to RLHF Alignment for Agents, the breakthrough of RLVR reasoning models (DeepSeek-R1), and the emergence of Tool-Grounded Post-Training via Agent RL (WebRL, MiRA).