Alert-list concordance exposes hidden divergence across clinical ICU imputation strategies

Comparing nine imputation methods on 40,336 ICU records reveals that pipelines with indistinguishable AUROC produce divergent high-risk patient alert lists, with Jaccard overlaps as low as 0.518.

Primary source: arXiv:2610.22635: https://arxiv.org/abs/2610.22635

What’s new: In clinical machine learning, preprocessing and missing-data imputation are known to influence downstream model behavior, yet imputation quality is almost universally audited using aggregate reconstruction metrics or population-level discrimination such as AUROC. However, aggregate metrics fail to indicate whether two pipelines flag the same individual patients for urgent clinical intervention. In Alert-List Concordance for Evaluating Imputation Strategies in ICU Sepsis Prediction (arXiv:2610.22635, 15 Oct 2026), researchers Priyanka Siddappa, Muntaqim Ahmed Raju, Shifat Haider Al Amin, and Ruizhe Ma introduce alert-list concordance, an evaluation paradigm that quantifies patient-level agreement between clinical pipelines using the Jaccard similarity of their top-k% alert sets accompanied by patient-level bootstrap confidence intervals.

The authors benchmarked nine imputation strategies alongside a native-missingness baseline using the PhysioNet/CinC 2019 challenge dataset, comprising 40,336 ICU patients and an average missingness rate of 71.9%. Each imputation method was evaluated with a fixed, Platt-calibrated XGBoost classifier. Following Bonferroni correction, 22 of the 28 pairwise comparisons between reconstruction-based imputation methods showed no statistically significant difference in patient-level AUROC. Despite this apparent aggregate equivalence, their actual alert lists overlapped at modest Jaccard similarities ranging from only 0.518 to 0.750, proving that identical aggregate metrics conceal substantial disagreement regarding which patients receive care alerts. ForwardFill emerged as a prominent outlier: while achieving marginally higher discrimination, it exhibited the lowest alert-list overlap in the entire matrix (0.40 to 0.45). This discrepancy persisted across independent hospital systems and across six distinct temporal aggregation rules. Furthermore, complex deep learning imputers failed to outperform simple mean imputation in discrimination while incurring 5 to 12 times greater computational wall-clock runtime.

Why it matters: Sepsis early-warning systems directly influence bedside triage and clinical resource allocation. If different preprocessing choices alter up to half of the flagged patient roster while maintaining identical AUROC statistics, model validation based solely on traditional metrics creates an illusion of clinical interchangeability. Alert-list concordance provides a vital reliability metric that reveals the clinical consequences of preprocessing choices prior to hospital deployment.

Caveats: The evaluation is conducted on retrospective electronic health record data within a fixed 1-hour discretization window, which may not capture rapid physiological fluctuations occurring between discrete chartings. The benchmark focuses on XGBoost tabular modeling for sepsis onset; deep sequence models (such as recurrent networks or state-space models) with learned temporal decay may exhibit different sensitivity profiles to imputation artifacts.