One post tagged with "train-time-scaling"

Welcome to the sixth installment of our definitive engineering breakdown of Stanford University's CS329A: Self-Improving AI Agents (Autumn 2025).

In Lectures 1 through 5 (Part 1: Foundation), we examined how agents push the limits of frozen, pre-trained models by allocating test-time compute: exploring search trees with MCTS, gathering runtime compiler diagnostics, and pruning dead ends.

Yet, we must confront a sober mathematical reality: If an elementary school student has never been taught the principles of calculus, giving them ten additional minutes to think during an exam will not produce a valid integration proof. Similarly, if an autoregressive model lacks the foundational representations within its frozen synaptic weights, sampling 10,000 rollouts or executing greedy beam search does not yield genuine intelligence—it produces stochastic luck.

Today marks the launch of Part 2: Training & Evolution (Lectures 6–9). We pivot from inference heuristics to fundamental neural weight updates:

"Instead of burning inference compute at runtime, how can we invest massive compute during training time to permanently rewrite the neural synapses of the model into an elite reasoner?"

In Lecture 6, we dismantle the core mathematical engine driving frontier reasoning models: Post-Training Reinforcement Learning (RL). We analyze three foundational paradigms that eliminated human labeling bottlenecks and scaled mathematical reasoning:

1. Autonomous Reasoning Bootstrapping: Self-Taught Reasoner (STaR, Zelikman et al., Stanford 2022).
2. The Death of the Critic: Group Relative Policy Optimization (GRPO, DeepSeekMath 2024).
3. Scaling to 1,000 GPUs: Decoupled Clipping and Dynamic Sampling (DAPO, 2025).

flowchart TD
    subgraph ParadigmShift ["The Architectural Paradigm Shift"]
        direction TB
        TestTime["Part 1: Test-Time Scaling<br/>• Frozen Neural Weights (θ = constant)<br/>• External Heuristics (MCTS, ReAct, RLEF)<br/>• Memory & Latency Overhead at Runtime"]
        --> TrainTime["Part 2: Train-Time Scaling (RL)<br/>• Active Weight Gradient Optimization (θ ← θ + Δθ)<br/>• Internalized Reasoning Instincts via RL<br/>• Fast, Single-Pass Inference Execution"]
    end

    subgraph RLEvolution ["The Evolution of Mathematical RL"]
        direction TB
        M1["STaR (Stanford 2022)<br/>• Bootstrapping from model's own correct rationales<br/>• Rationalization: Generating backward hints during training<br/>• 6B model matches 175B GPT-3 on CommonsenseQA"]
        --> M2["GRPO (DeepSeekMath 2024)<br/>• Completely removes Critic model (Cuts GPU VRAM by 50%)<br/>• Intra-group Advantage normalization across G rollouts<br/>• Pure 7B model beats Google Minerva 540B on MATH"]
        --> M3["DAPO (Industrial Open-Source 2025)<br/>• Clip-Higher (0.28 vs 0.20) prevents Entropy Collapse<br/>• Dynamic Sampling filters zero-variance batches<br/>• Qwen-32B breaks AIME 2024 record in 50% fewer steps"]
    end

    ParadigmShift ==> RLEvolution