Predictable performance is suboptimal: Auxiliary routing predictors fall short of model self-confidence

Mathematical analysis proves that auxiliary performance predictors outperform an LLM's internal confidence only when candidate models are suboptimal, revealing structural ceilings for model cascading.

Primary source: arXiv:2610.31428: https://arxiv.org/abs/2610.31428

What’s new: Production large language model deployments increasingly employ adaptive routing and cascading architectures, dynamically routing simpler queries to low-cost small models while escalating complex prompts to frontier models. Existing routing systems typically train separate, auxiliary performance predictors (such as classification heads or external surrogate models) to forecast whether a specific candidate model will answer a given query correctly. In Predictable performance is suboptimal, and what it means for model routing (arXiv:2610.31428, 22 Oct 2026), researchers Guanhua Zhang and Moritz Hardt from the Max Planck Institute for Intelligent Systems establish theoretical limits on when performance prediction can outperform simple baseline routing strategies.

The authors demonstrate that auxiliary performance predictors consistently fail to beat an elementary baseline: the candidate model’s own self-confidence, defined as the token probability the model assigns to its generated answer. Through rigorous mathematical analysis, Zhang and Hardt prove that any performance predictor’s advantage over internal self-confidence is strictly bounded by the candidate model’s suboptimality on the task, scaled by the additional capacity introduced by the predictor. When an underlying model is well-calibrated or close to optimal within its capacity class, an external predictor cannot reliably forecast errors without possessing substantial superior predictive capacity over the task domain itself. Empirical evaluations across language benchmarks confirm that gains from complex routing predictors decay sharply as models undergo alignment and calibration.

Why it matters: Engineering teams often invest significant computational resources into designing, training, and maintaining complex auxiliary routing models and embedding classifiers to optimize inference costs. This work demonstrates that external routing networks offer diminishing returns over simple thresholding on model self-confidence. Improving candidate model calibration and training models to report accurate epistemic certainty yields superior routing decisions compared to introducing external prediction overhead.

Caveats: The theoretical bounds assume standard cross-entropy and scoring formulations where candidate model probabilities can be accessed directly at test time. For closed-source commercial APIs that obfuscate token log-probabilities or return normalized scores without full output distributions, external surrogate predictors may remain practically necessary despite theoretical suboptimality.