Frozen site-probe accuracy fails as a standalone score for multi-site MRI harmonization

An audit across 1,112 ABIDE I brain scans and seven harmonization methods shows frozen site classifiers swing by 46 points across seeds, fall to chance under simple blur, and miss residual scanner signals.

Primary source: arXiv:2610.14140: https://arxiv.org/abs/2610.14140

What’s new: Image-level MRI harmonization aims to remove scanner-specific contrast variations across clinical sites so pooled cohorts can be analyzed without domain shift. Because paired traveling-subject scans are rare, studies routinely validate harmonization indirectly using a frozen site probe: a classifier trained once on raw scans and applied unchanged to harmonized outputs, where a drop in site-prediction accuracy is interpreted as successful removal of scanner effects. In Frozen Site-Probe Accuracy Is Not a Standalone Harmonization Score: An Audit of Evaluation Practice for Multi-Site MRI (arXiv:2610.14140, 9 Oct 2026), researchers from Universidad Autonoma de Madrid and Universite de Bordeaux audit this evaluation protocol on 1,112 T1-weighted scans across 17 sites from ABIDE I, testing ten output sets from seven harmonization methods alongside explicit controls for head geometry, intensity histograms, preprocessing, and target alignment.

The audit exposes three distinct failure modes. First, the frozen probe verdict is highly sensitive to random initialization: among probes that all classify raw scans accurately, balanced accuracy on the exact same CycleGAN outputs ranges from 0.13 to 0.59 across five seeds (chance is 0.06), and training probes longer to convergence pushes aggressive StarGAN scores from 0.21 to 0.70. Second, applying a simple Gaussian blur that moves image intensities away from the NYU target site drops frozen-probe accuracy to 0.07 (chance level) merely by presenting unfamiliar textures. Third, low frozen-probe scores fail to certify that site signatures were removed: on pretrained HACA3 outputs, the frozen probe scores 0.25, yet a fresh probe trained directly on the harmonized outputs recovers the acquisition site at 0.75 balanced accuracy, and a probe using only intensity histograms reaches 0.71.

Why it matters: Multi-site neuroimaging pipelines depend on harmonization metrics to verify that scanner confounders are eliminated before training downstream diagnostic models. Demonstrating that a single frozen classifier conflates arbitrary distribution shift with site invariance invalidates a common benchmark shortcut. The authors replace single-number probe reporting with a rigorous evaluation checklist combining multi-seed converged probes, output-trained retraining probes, brain-only intensity controls, and direct Wasserstein target alignment.

Caveats: The study evaluates single 2D axial slices per subject on one T1-weighted cohort (ABIDE I) aligned to a single reference site (NYU), with 2 to 15 test subjects per source site. Apart from pretrained HACA3, the harmonization baselines are untuned implementations built to illustrate evaluation mechanics rather than rank state-of-the-art harmonizers. Because site labels in retrospective cohorts are entangled with patient demographics and head shape, even retrained probes cannot completely separate scanner contrast effects from underlying biological differences.