Skip to main content

Stanford CS329A (Ep 12): Agent Evaluation & Long Tasks — METR Time Horizons, OpenAI GDPval, and DeepScholar-Bench

· 11 min read
Jaybee
Jaybee
AI & Robotics Engineer

If an engineering team builds an autonomous coding agent and presents it to enterprise executives or investors, quoting a "92% score on MMLU" yields blank stares. The immediate business questions are vastly more concrete:

  • "Can this autonomous agent handle a multi-hour, end-to-end refactoring ticket without human hand-holding?"
  • "If a senior human engineer takes 8 hours to construct this financial model or debug this distributed microservice, can the agent deliver comparable quality? And what is its net economic value in real dollars?"

Classic academic benchmarks (MMLU, GSM8K, HumanEval) are thoroughly saturated and compromised by internet data contamination. More fundamentally, scoring well on 5-second multiple-choice trivia is completely decoupled from executing Long-Horizon Agent Tasks requiring hundreds of interdependent, stateful actions over multiple hours.

In this grand finale to Part 3: Agent Engineering (Lectures 10 through 12) of Stanford CS329A: Self-Improving AI Agents, we examine the modern science of Agent Evaluation Benchmarks — measuring agentic intelligence not through arbitrary percentages, but through physical units of Human Time and Economic Dollars ($):

  • Human Time Horizon Scaling (METR, 2025): The 50% success time horizon (h_agent) and the empirical 7-month doubling law.
  • Real-World Economic Value (OpenAI GDPval, 2025): Evaluating 44 occupations across 9 US GDP sectors and the rigorous mathematics of the "Try-n-then-fix" human review penalty.
  • Uncontaminated Academic Research Evaluation (DeepScholar-Bench, 2025): 3-dimensional 7-metric live paper analysis and position-bias debiasing for LLM judges.

Stanford CS329A Lecture 12: Agent Evaluation Benchmarks, METR, GDPval, and DeepScholar-Bench

1. The Evaluation Crisis: The Collapse of Multiple-Choice Metrics​

For half a decade, the AI community relied on static question-answering benchmarks. However, evaluating autonomous agents exposed fatal structural breakdowns in this methodology:

  1. Saturation & Contamination: Frontier models quickly exceeded 90% on MMLU. Because benchmark questions were scraped by web crawlers during pre-training, high scores often reflected rote memorization rather than generalized reasoning.
  2. Zero Temporal Horizon: Answering a 4-choice question takes 2 seconds and 1 step. Production engineering demands maintaining coherent internal state across 100+ sequential tool calls over hours.
  3. The Cleanliness Illusion: Traditional tests present clean, well-specified prompts. Real-world tasks are inherently "messy" — involving ambiguous requirements, dynamic environment shifts, uninformative error logs, and irreversible execution hazards.

To evaluate real-world utility, frontier AI labs have abandoned multiple-choice exams in favor of rigorous, continuous physical metrics.

┌─────────────────────────────────────────────────────────────────────────┐
│               The 3 Pillars of Modern Agent Evaluation                  │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
       ┌─────────────────────────────┼─────────────────────────────┐
       ▼                             ▼                             ▼
┌──────────────────────────┐  ┌──────────────────────────┐  ┌──────────────────────────┐
│   METR Time Horizons     │  │   OpenAI GDPval ($)      │  │   DeepScholar-Bench      │
├──────────────────────────┤  ├──────────────────────────┤  ├──────────────────────────┤
│ • Unit: Human Time (t)   │  │ • Unit: Real US Dollars  │  │ • Unit: 7-Metric Vector  │
│ • 50% Success Horizon h  │  │ • 44 Professional Roles  │  │ • Live arXiv Ingestion   │
│ • 7-Month Doubling Law   │  │ • Try-n-then-fix Formula │  │ • Zero Contamination     │
│ • Task Messiness Penalty │  │ • 14-Yr Veteran Blind J. │  │ • Nugget Coverage (39%)  │
│ • 2s (GPT-2) -> 110m (o3)│  │ • Win Rate Threshold 40% │  │ • Swapped-Order Debiasing│
└──────────────────────────┘  └──────────────────────────┘  └──────────────────────────┘

2. Measuring Intelligence in Human Time: METR (2025)​

Stating that a model achieves an 85% score provides zero intuition regarding how long it can operate autonomously. The non-profit research lab METR (Model Evaluation and Threat Research) revolutionized agent benchmarking by standardizing capability along the axis of human time:

"Discard percentage scores. Instead, measure the probability that an AI agent successfully completes a task that takes a skilled human expert time $t$. Define the agent's intelligence by its 50% Time Horizon (h_agent) — the duration of a human task it completes with a 50% success rate."

    [170 Complex Software / ML Engineering Tasks (Seconds to 8 Hours)]
                               │
                ┌──────────────┴──────────────┐
                ▼                             ▼
       [Human Expert Time t]          [Agent Empirical Pass Rate p]
                │                             │
                └──────────────┬──────────────┘
                               ▼
        [Item Response Theory (IRT) Logistic Regression Curve]
           p = σ( (log h_agent - log t_task) × β_agent )
                               │
                               ▼
    [Agent 50% Time Horizon h_agent: GPT-2 (2s) ──► OpenAI o3 (110m)]

The Logistic Time-Horizon Formulation​

Drawing from psychometric Item Response Theory (IRT), METR models an agent's completion probability $p$ as a logistic function of human task duration t_task and agent capacity h_agent:

========================================================================================
METR Time-Horizon Completion Probability Formulation:
========================================================================================
p_success(agent, task) = σ( (ln(h_agent) - ln(t_task)) * β_agent )

Where:
  • t_task  : Average time required by a vetted human expert to complete the task
  • h_agent : The agent's 50% success time horizon (when t_task = h_agent, p = 0.5)
  • β_agent : Slope parameter indicating performance decay as task length increases
  • σ(z)    : Standard logistic sigmoid 1 / (1 + exp(-z))
========================================================================================

The 7-Month Doubling Law​

Evaluating 16 frontier models spanning 2019 to 2025 revealed a breathtaking, deterministic exponential scaling trend:

  • 2019 (GPT-2): Time horizon of 2 seconds (simple autocomplete).
  • 2021 (GPT-3): Time horizon of 15 seconds.
  • 2023 (GPT-4): Time horizon of 10 minutes.
  • 2025 (OpenAI o3): Time horizon of 110 minutes (~2 hours).
  • The Empirical Trendline: Frontier AI time horizons double exactly every 7 months (207 days).
  • Macroscopic Projection: Maintaining this empirical trajectory indicates that between 2028 and 2030, frontier autonomous agents will achieve a 50% success horizon on tasks requiring one full month (160 hours) of continuous human expert labor.

3. Measuring Real Economic Value: OpenAI GDPval (2025)​

While METR calibrated intelligence in units of time, OpenAI’s GDPval benchmark (released in late 2025) grounded agent capability in cold, hard capital.

OpenAI selected 44 high-earning digital occupations contributing to the 9 major sectors of US Gross Domestic Product (finance, legal, software, engineering, healthcare administration, etc.). They hired vetted professionals with an average of 14 years of industry experience to craft 220 representative, high-stakes tasks (e.g., building discounted cash flow models, drafting patent briefs, and writing data analysis pipelines):

  • Average human completion time: 9.5 hours (median 5.0 hours).
  • Average market cost per task: $398 USD (~550,000 KRW).

Blind Win Rates against Human Veterans​

Senior industry veterans evaluated agent outputs against human peer submissions in double-blind comparative assessments:

  • GPT-4o: Blind Win Rate of 12.5% (easily distinguished and rejected by senior humans).
  • OpenAI o3: Blind Win Rate of 35.2%.
  • Claude Opus 4.1: Blind Win Rate of 47.6% (Won or Tied)!
  • Frontier reasoning agents now produce deliverables that experienced professionals judge as equal or superior to human peer work nearly half the time.

The "Try-n-then-fix" Review Penalty Equation​

Marketing claims frequently tout: "The agent finished a 9-hour task in 2 minutes, delivering a 270x speedup!" GDPval disproves this naive view by formalizing the Human Verification and Rework Penalty:

========================================================================================
The Realistic Economic Time-Cost Equation (Try-n-then-fix):
========================================================================================
E[T_real] = (M_T + R_T) * [ (1 - (1 - w)^n) / w ] + (1 - w)^n * H_T

Where:
  • M_T : Machine generation time (minutes)
  • R_T : Human expert review time required to audit the machine output
  • H_T : Human baseline time to complete the entire task from scratch
  • w   : Model Win Rate (Probability that the generated draft is acceptable)
  • n   : Maximum allowed agent retries before human abandons and takes over
========================================================================================

The Economic Threshold: Deficit vs Surplus​

The mathematical implications of this equation are striking:

  • When agent win rate is low ($w < 20%$, e.g., GPT-4o), human experts spend hours reviewing flawed drafts, get frustrated by subtle hallucinations, and ultimately discard the work to restart from scratch. Consequently, deploying GPT-4o yielded an overall net slowdown (0.87x human baseline speed).
  • Only when win rate crosses the critical threshold of $w \ge 40%$ (achieved by o3 and Opus 4.1) does the human review penalty become economically viable, yielding an authentic 1.3x to 1.4x net productivity acceleration.

4. Live Academic Research Evaluation: DeepScholar-Bench (2025)​

How do we evaluate autonomous agents on open-ended creative and scholarly tasks where no single ground-truth answer exists?

Released in August 2025, DeepScholar-Bench established a continuous, live evaluation pipeline ingesting newly published papers directly from arXiv every week, eliminating data contamination entirely.

       [Live Weekly arXiv Ingestion (Zero Contamination Guarantee)]
                               │
                               ▼
┌─────────────────────────────────────────────────────────────────────────┐
│               DeepScholar-Bench 3-Dimensional 7-Metric Stack             │
├─────────────────────────────────────────────────────────────────────────┤
│ 1. Knowledge Synthesis (너깃 종합력)                                     │
│    • Organization & Coherence (Logical structure, flow, formatting)     │
│    • Nugget Coverage (Recall of atomic factual units from ground truth) │
│ 2. Retrieval Quality (검색 정밀도)                                      │
│    • Relevance Rate (Topical proximity of cited literature)             │
│    • Document Importance (Citation of authoritative seminal papers)     │
│ 3. Verifiability (학술적 검증 가능성)                                   │
│    • Citation Precision (Rigorous verification against primary sources) │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
        [Swapped-Order LLM Judge: Debiasing Position Preference]

The 30.9% Frontier Reality Check​

Despite glowing press releases regarding autonomous AI researchers, DeepScholar-Bench revealed that even frontier systems like OpenAI DeepResearch achieved a geometric mean score of only 30.9% across the 7 metrics:

  • Surface Eloquence vs Substantive Coverage: Agents scored high on writing style and formatting (Organization: 85.7%), but stumbled severely on capturing vital facts (Nugget Coverage: 39.2%) and isolating landmark literature (Document Importance: 12.4%).
  • The Oracle Retriever Experiment: When researchers fed the agent the exact gold-standard paper list (eliminating search errors entirely), overall performance only rose to 49%. This proves that the fundamental bottleneck in autonomous research is not information retrieval, but deep conceptual synthesis.

5. Comparative Matrix of Modern Agent Evaluation Benchmarks​

=============================================================================================================
Benchmark             METR (2025)                   OpenAI GDPval (2025)          DeepScholar-Bench (2025)
-------------------------------------------------------------------------------------------------------------
Primary Measurement   Human Time Horizon (Hours)    Net Economic Value ($ USD)    Live Multi-Metric Quality
Evaluation Target     Full-Stack Software/ML Tasks  44 White-Collar Professions   Open-Ended Literature Review
Ground Truth Type     Deterministic Test Suites     14-Yr Human Blind Win Rate    Atomic Information Nuggets
Contamination Defense Private Air-Gapped Tasks      Proprietary Enterprise PRDs   Continuous Weekly arXiv Feed
Human Review Cost     Not Modeled (Automated Pass)  Try-n-then-fix Penalty Equat. Swapped-Order Judge Debias
Current Frontier Peak o3: 110 Minutes (50% Pass)    Opus 4.1: 47.6% Blind Win RateDeepResearch: 30.9% Geo-Mean
=============================================================================================================

6. Deep Technical Q&A​

Q1: What constitutes "Task Messiness" in METR, and how does it degrade agent performance?​

In METR’s formal framework, Task Messiness quantifies 16 real-world environmental friction factors absent in clean coding puzzles:

  • Dynamic State: The environment mutates while the agent reasons (e.g., live network latencies, background database locks).
  • Irreversible Actions: Destructive operations (e.g., dropping a table or pushing broken git tags) where mistakes cannot be undone by simple backtracking.
  • Active Discovery Requirements: The agent is not handed inputs directly; it must search log directories, issue grep commands, and infer context.

Empirical regression showed that for every 1-point increase on METR’s task messiness scale, agent success rate dropped by 8.1%. Real-world engineering superiority is governed by resilience to environmental noise, not raw algorithmic syntax generation.

Q2: How does "Swapped-Order Averaging" eliminate Position Bias in LLM Judges?​

When an LLM judge compares two candidate outputs $(A)$ and $(B)$, it exhibits a pervasive Position Bias, favoring candidate $(A)$ merely because it appears first in the prompt sequence (often with a 60%~70% baseline preference).

To mathematically cancel this systemic skew, DeepScholar-Bench implements Swapped-Order Averaging:

  1. Run Pass 1: Present options as $(A, B)$ and record score vector $S_1 = [s(A), s(B)]$.
  2. Run Pass 2: Invert sequence order as $(B, A)$ and record score vector $S_2 = [s(B), s(A)]$.
  3. Compute the unweighted arithmetic mean:
S_final(A) = (S_1(A) + S_2(A)) / 2,    S_final(B) = (S_1(B) + S_2(B)) / 2

By evaluating both permutations under identical temperature parameters, position preference cancels out algebraically, yielding a robust, unbiased comparative signal.

Conclusion & Part 3 Wrap-Up​

Part 3 (Agent Engineering) has systematically dismantled the illusions surrounding autonomous systems:

  • In Lecture 10, we established real-world software repair via SWE-Agents and profiler-in-the-loop KernelBench GPU tuning.
  • In Lecture 11, we conquered long-term persistence using MemGPT, Cartridges, and CacheBlend.
  • In Lecture 12, we grounded agent utility in the unforgiving physical realities of Human Time (METR) and Economic Value (GDPval).

We now possess the complete systems engineering stack required to construct, deploy, optimize, and measure self-improving agents.

Looking Ahead to Part 4: The Frontiers​

In our final section, Part 4 · Frontiers (Lectures 13 through 17), we venture beyond systems engineering into the theoretical apex of artificial intelligence: The Nature of Machine Reasoning.

In Lecture 13, we launch Part 4 with: 《LLM Reasoning: The Foundations of Machine Reasoning, Chain-of-Thought, and Emergent Cognitive Scaling》, dissecting seminal breakthroughs by Google DeepMind’s Denny Zhou.

Key Takeaways​

  • Measure Time, Not Trivia: The true metric of agent intelligence is the 50% success time horizon (h_agent), which has doubled every 7 months from 2 seconds (GPT-2) to 110 minutes (o3).
  • The Review Penalty Governs ROI: Unless an agent exceeds a ~40% blind win rate, human audit and rework penalties result in a net economic loss.
  • Synthesis is the Frontier: In open-ended research (DeepScholar-Bench), information retrieval is largely solved; synthesizing scattered literature into coherent, verified insights remains the core bottleneck.
ADVERTISEMENT