Selective auditing guarantees bounded AI error rates with minimal human supervision

Gibbs, Han, Tibshirani, and Zrnic develop an online selective auditing framework that provably bounds unaudited AI decision errors below user tolerances under both i.i.d. and adversarial streams.

Primary source: arXiv:2610.31662: https://arxiv.org/abs/2610.31662

What’s new: Autonomous AI systems in production regularly evaluate high-stakes decisions, accompanied by confidence estimates from internal heads or external verifiers. Deploying human oversight for every decision is economically unfeasible, yet trusting model confidence scores blindly risks letting catastrophic mistakes slip through unnoticed. In Reliable AI-Assisted Decisions via Selective Auditing (arXiv:2610.31662, 22 Oct 2026), researchers Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani, and Tijana Zrnic from UC Berkeley and Stanford formulate selective auditing as an online constrained optimization framework with distribution-free theoretical guarantees.

The goal of selective auditing is to minimize expert human auditing frequency while guaranteeing that the long-run fraction of AI errors that bypass auditing remains strictly below a user-specified tolerance. The authors develop a family of online auditing algorithms that require zero distributional assumptions, holding under both independent and identically distributed (i.i.d.) streams and non-stationary adversarial sequences. Among these, they present an asymptotically optimal method that achieves the minimax lower bound on necessary auditing rates. Tested on production-style tasks including safety screening for harmful LLM prompts, qualitative text annotation, and medical data labeling, the algorithm maintains valid error control while significantly cutting human auditing load compared to standard fixed-threshold and conformal baselines.

Why it matters: Most AI governance frameworks demand human-in-the-loop oversight for safety-critical deployments, but static thresholding cannot handle distribution shift, model miscalibration, or adversarial evasion. Selective auditing provides mathematical guarantees that allow organizations to deploy agentic workflows and automated triaging with certified safety bounds, dynamically scaling human oversight up or down in response to incoming task difficulty.

Caveats: The framework assumes that expert auditors are infallible or near-infallible ground-truth arbiters who reliably detect and correct AI mistakes upon inspection. In real-world settings where human reviewers suffer from cognitive fatigue, subjective disagreement, or domain ambiguity, audit errors could degrade the empirical reliability of the mathematical guarantee.