The predictability bias: why JEPA-style pretraining under-represents pathology in CT

TAP-JEPA, a ViT-B/3D trained on 113,611 CT volumes, is audited to show that masked prediction learns anatomy well but pulls sporadic pathology toward normal.

Primary source: arXiv:2610.13237: https://arxiv.org/abs/2610.13237

What’s new: Joint-embedding predictive architectures (JEPAs) learn by predicting teacher representations of masked regions from visible context. In The Predictability Bias of Joint-Embedding Predictive Learning in Medical Imaging (arXiv:2610.13237, 9 Oct 2026), the authors argue that point prediction can only capture the part of a target that the context resolves. They call this constraint the predictability bias. In CT, recurrent anatomy is predictable from surrounding tissue, while the sporadic, localized pathology that drives clinical interpretation often is not.

The authors train TAP-JEPA, a ViT-B/3D on 113,611 CT volumes, that is competitive with COLIPRI, Merlin, and VoxelFM on scan-level classification. They then audit the predictive pathway by replaying the exact masking, teacher, and predictor computations used in training. Prediction lowers mean Skill (prevalence-normalized average precision) from 0.388 to 0.271 for pathology, but only from 0.921 to 0.873 for anatomy. Along the teacher direction, prediction moves pathology representations 34.7% toward matched controls from the same organ, and moves controls 13.6% toward pathology, with larger shifts farther from visible context.

Why it matters: Competitive downstream scores can hide what a pretraining objective discards. This work gives a concrete audit recipe for medical self-supervised learning, and the finding that contextual predictability is not a sufficient proxy for clinical relevance should inform masking strategies and objectives for any imaging foundation model.

Caveats: The analysis is on CT and one architecture family, so it is not established that other modalities or objectives behave the same way. Removing all final-layer pathology tokens reduced the decision margin by at most 0.45% in four of seven scan-level probes, which suggests the downstream impact of the bias is real but modest on the current probe suite. The paper diagnoses the problem and does not yet demonstrate a fix.