2 posts tagged with "SWE-bench"

If an engineering team builds an autonomous coding agent and presents it to enterprise executives or investors, quoting a "92% score on MMLU" yields blank stares. The immediate business questions are vastly more concrete:

• "Can this autonomous agent handle a multi-hour, end-to-end refactoring ticket without human hand-holding?"
• "If a senior human engineer takes 8 hours to construct this financial model or debug this distributed microservice, can the agent deliver comparable quality? And what is its net economic value in real dollars?"

Classic academic benchmarks (MMLU, GSM8K, HumanEval) are thoroughly saturated and compromised by internet data contamination. More fundamentally, scoring well on 5-second multiple-choice trivia is completely decoupled from executing Long-Horizon Agent Tasks requiring hundreds of interdependent, stateful actions over multiple hours.

In this grand finale to Part 3: Agent Engineering (Lectures 10 through 12) of Stanford CS329A: Self-Improving AI Agents, we examine the modern science of Agent Evaluation Benchmarks — measuring agentic intelligence not through arbitrary percentages, but through physical units of Human Time and Economic Dollars ($):

• Human Time Horizon Scaling (METR, 2025): The 50% success time horizon (h_agent) and the empirical 7-month doubling law.
• Real-World Economic Value (OpenAI GDPval, 2025): Evaluating 44 occupations across 9 US GDP sectors and the rigorous mathematics of the "Try-n-then-fix" human review penalty.
• Uncontaminated Academic Research Evaluation (DeepScholar-Bench, 2025): 3-dimensional 7-metric live paper analysis and position-bias debiasing for LLM judges.

Real-world production software engineering bears virtually no resemblance to isolated competitive programming puzzles or academic toy problems. A single GitHub bug report does not arrive neatly packaged with pre-extracted unit tests; it is entangled across hundreds of files, legacy dependencies, and millions of lines of distributed production code. Furthermore, software in production cannot merely be functionally correct — it must be computationally performant, memory-efficient, and tightly aligned with low-level parallel hardware architectures.

In this kickoff to Part 3: Agent Engineering (Lectures 10 through 12) of Stanford CS329A: Self-Improving AI Agents, we transition from fundamental model training clusters into the engineering trenches of production software development:

"How do autonomous AI agents navigate multi-million-token enterprise repositories, isolate elusive regression bugs, synthesize reproducible test harnesses, and write custom GPU kernels that out-accelerate human-crafted PyTorch baselines?"

This post investigates the modern frontier of SWE-Agent Autonomous Coding, exploring three landmark breakthroughs: CodeMonkeys Automated Refactoring on the SWE-bench Benchmark GitHub, KernelBench GPU Performance Tuning using hardware profiler loops, and Stanford’s Agent-Computer Interface ACI (Agent-System Interface, ASI) designed to eliminate semantic misalignment between high-level agent intent and physical parallel hardware.