Stanford CS329A (Ep 12): Agent Evaluation & Long Tasks — METR Time Horizons, OpenAI GDPval, and DeepScholar-Bench
If an engineering team builds an autonomous coding agent and presents it to enterprise executives or investors, quoting a "92% score on MMLU" yields blank stares. The immediate business questions are vastly more concrete:
- "Can this autonomous agent handle a multi-hour, end-to-end refactoring ticket without human hand-holding?"
- "If a senior human engineer takes 8 hours to construct this financial model or debug this distributed microservice, can the agent deliver comparable quality? And what is its net economic value in real dollars?"
Classic academic benchmarks (MMLU, GSM8K, HumanEval) are thoroughly saturated and compromised by internet data contamination. More fundamentally, scoring well on 5-second multiple-choice trivia is completely decoupled from executing Long-Horizon Agent Tasks requiring hundreds of interdependent, stateful actions over multiple hours.
In this grand finale to Part 3: Agent Engineering (Lectures 10 through 12) of Stanford CS329A: Self-Improving AI Agents, we examine the modern science of Agent Evaluation Benchmarks — measuring agentic intelligence not through arbitrary percentages, but through physical units of Human Time and Economic Dollars ($):
- Human Time Horizon Scaling (METR, 2025): The 50% success time horizon (h_agent) and the empirical 7-month doubling law.
- Real-World Economic Value (OpenAI GDPval, 2025): Evaluating 44 occupations across 9 US GDP sectors and the rigorous mathematics of the "Try-n-then-fix" human review penalty.
- Uncontaminated Academic Research Evaluation (DeepScholar-Bench, 2025): 3-dimensional 7-metric live paper analysis and position-bias debiasing for LLM judges.