Auditing open LLMs on the Spanish MIR exam exposes uncalibrated abstention and image blindness

Evaluating 8B-27B open models on 1,552 official residency exam questions shows that while raw scores qualify models for 43 specialties, medical fine-tuning harms calibration and multimodal models ignore clinical images.

Primary source: arXiv:2610.31586: https://arxiv.org/abs/2610.31586

What’s new: High accuracy on standardized medical licensing exams is frequently cited as proof of clinical readiness, but healthcare systems seeking to deploy open-weight models on local hospital servers require deeper insight into calibration, reasoning fidelity, and multimodal competence. In Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination (arXiv:2610.31586, 22 Oct 2026), researchers Miguel Fernandez-de-Retana, Unai Zulaika, Ruben Sanchez-Corcuera, Aritz Bilbao-Jayo, and Aitor Almeida evaluate four open models ranging from 8B to 27B parameters on 1,552 questions across eight annual editions of the Spanish Medical Intern Resident (MIR) examination.

Scored under the MIR’s official rules (where incorrect choices incur a negative marking penalty of -0.33 points), the three largest models rank between the top 600th and 3,000th candidates nationally, achieving marks sufficient to enter 41 to 43 out of 46 medical specialties. However, detailed probing reveals three critical failure modes. First, selective abstention under penalty rules never improves rank: because confidence scores are poorly calibrated, even the models’ lowest-confidence answers exceed the 25% random-guess threshold. Second, domain-specific medical fine-tuning reduced raw overconfidence but lowered overall accuracy and failed to improve confidence separation between correct and incorrect predictions. Third, in 13% to 38% of correct answers, models dismissed incorrect options using medically unsound reasoning that was invisible in confidence scores. Crucially, ablation experiments revealed that removing or randomly shuffling provided clinical images (radiology scans, histology slides, ECG traces) produced zero change in accuracy, proving that multimodal reasoning was entirely bypassed in favor of text priors.

Why it matters: Hospital administrators and clinical engineers often rely on aggregate exam benchmark scores when evaluating whether open models are safe for on-premise clinical assistance. This study demonstrates that high benchmark rankings can mask severe clinical deficits: models cannot gauge their own ignorance, reach correct conclusions through invalid diagnostic rationales, and fail to ground answers in provided diagnostic images. Clinical deployment evaluations must audit counterfactual image grounding and rationale soundness rather than relying on headline accuracy.

Caveats: The study focuses on multiple-choice questions from the Spanish MIR exam, which reflect standardized European medical curricula but do not capture unstructured physician-patient dialogues or continuous EHR timelines. In addition, the evaluation was conducted on 8B to 27B parameter open models; whether 70B+ open models or proprietary frontier models exhibit similar image insensitivity on the MIR benchmark remains to be determined.