AI time horizons: a statistical audit of the METR scaling plot

Nguyen and Fithian re-evaluate METR’s 50% time horizon metric with splines and item-response theory, revealing that task difficulty does not scale linearly with human log-time.

Primary source: arXiv:2610.12466: https://arxiv.org/abs/2610.12466

What’s new: METR’s 50% time horizon has become a benchmark for autonomous AI capability, measuring the human completion time of software tasks that an AI system solves with 50% probability. In On the estimation and validity of AI time horizons: a statistical look at the METR plot (arXiv:2610.12466, 8 Oct 2026), UC Berkeley statisticians Drew T. Nguyen and William Fithian conduct a rigorous psychometric and statistical audit of this metric across 228 software engineering tasks and 26 frontier AI systems.

The authors show that the original formulation implicitly assumes AI difficulty scales linearly with the logarithm of human completion time. By relaxing this constraint using splines and item-response theory (IRT), they fit an empirical transfer function mapping human duration to machine difficulty. The resulting curve exhibits strong non-linearities: it is nearly flat in the 2 to 30 minute range, but steepens sharply beyond 30 minutes. Consequently, an apparent capability jump from 3 minutes to 30 minutes requires far less algorithmic progress than jumping from 30 minutes to 5 hours, despite both spans representing an identical 10x multiplier in human time.

Why it matters: AI governance commitments, frontier safety policies, and capability projections frequently anchor on time horizons to establish dangerous capability thresholds. If a benchmark’s underlying metric conflates logarithmic human duration with machine difficulty, policy teams risk miscalculating doubling rates and deployment timelines. Nguyen and Fithian provide cross-validated scoring estimators and diagnostic construct-validity checks that give benchmark builders a principled foundation for longitudinal tracking.

Caveats: The empirical spline is derived from METR’s specific suite of 228 software engineering and cybersecurity challenges. Generalization across non-programming domains, wet-lab biology workflows, or subjective open-ended planning remains to be demonstrated. As benchmarks incorporate multi-day autonomous agent tasks, task variance and human completion noise will require even larger sample sizes.