Φ-Bench: can LLMs engineer the infrastructure that runs them?

An 85-task infra/eval benchmark spanning kernels, long-horizon repo work, and end-to-end optimization — topped by Claude Opus 5 at 36.53%, with a lot of headroom left.

Primary source: arXiv:2609.10226: https://arxiv.org/abs/2609.10226

What’s new: Kernel microbenchmarks and operator fill-in tasks do not answer whether models can do real LLM-infra engineering — navigating a training/serving codebase, implementing features across files, or chasing an end-to-end efficiency goal. Φ-Bench (arXiv:2609.10226, Ding et al., submitted 9 Sep 2026) is an 85-task Frontier AI Infrastructure Benchmark grounded in frontier research problems and real repositories.

Tasks come in three graduated formats: Kernel Function Completion (KFC) (55 tasks) — implement a specified function/interface and pass correctness plus numerical checks; Long-Horizon Implementation (LHI) (20) — implement a feature with the path left open across multiple files; End-to-End Optimization (E2EO) (10) — pursue a system-level objective with repository-wide changes. Scoring is correctness-gated (build, tests, edit constraints, anti-cheating) before any efficiency reward. On the leaderboard in the paper, Claude Opus 5 leads overall at 36.53%, then Kimi K3 (28.12%) and Qwen3.8 Max (27.73%); several other frontier models land between roughly 13% and 25%. Category scores vary widely (training and I/O look easier than hardware/edge in the reported table).

Why it matters: Infra is where “AI beyond chat” pays rent — kernels, serving stacks, parallelism, memory. Φ-Bench is useful because it measures open-ended engineering under a harness that tries to catch reward hacking, not just autocomplete of a fused op. The headline number is the gap: even the best model clears only about a third of the score, which is the honest product claim for autonomous infra agents right now.

Assets pointed from the paper include a leaderboard (faibench.org), GitHub, and a Hugging Face dataset — the right packaging if you want to reproduce rather than screenshot a bar chart.

Caveats: Preprint. Aggregate percentages mix binary implementation tasks and continuous efficiency tasks under the authors’ gating rules — compare models inside Φ-Bench, not against unrelated SWE or kernel benches without care. Scaffolding, iteration caps, and “max reasoning” settings matter; the paper itself studies iteration and budget effects. Anti-cheating helps, but any agent benchmark with visible tests can still be gamed in new ways. Read the 36.53% as “substantial headroom,” which is exactly how the authors frame it — not as a solved path to self-optimizing clusters.