PathLang: probing language sensitivity and semantic competition in pathology VLMs

Cheng et al. evaluate nine computational pathology vision-language models across pathologist-validated phrasing, uncovering high sensitivity to clinical paraphrases and diagnostic competition.

Primary source: arXiv:2610.11329: https://arxiv.org/abs/2610.11329

What’s new: Vision-language models (VLMs) in computational pathology are widely evaluated on zero-shot diagnostic classification, but standard benchmarks hold language fixed using rigid, canonical prompt templates. In routine hospital practice, pathologists express identical histological findings using varied terminology, different levels of diagnostic granularity, and institutional reporting styles. In PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology (arXiv:2610.11329, 8 Oct 2026), Cheng et al. introduce PathLang, a benchmark that systematically isolates the impact of diagnostic phrasing.

PathLang fixes histological slides, ground-truth labels, and multimodal evaluation tasks while varying only the language axis. All prompt variations, semantic-equivalence paraphrases, and candidate diagnosis pools were curated and verified by six board-certified pathologists. Spanning four task families across nine pathology VLMs and five public datasets from four organ systems, PathLang exposes substantial brittleness: models exhibit large accuracy drops when presented with clinically equivalent paraphrases. Furthermore, open-vocabulary retrieval tests demonstrate that high image-text alignment scores do not reliably ensure inter-class separability when candidate options feature fine-grained semantic competition.

Why it matters: A clinical AI system that correctly detects a lesion under one phrasing but fails when a physician enters an equivalent synonymous description cannot be safely deployed. PathLang highlights that robust medical vision-language evaluation requires testing across the natural linguistic distribution of clinical practice rather than relying on narrow template prompts.

Caveats: The benchmark evaluates zero-shot classification and cross-modal retrieval over frozen whole-slide and patch representations. It does not evaluate full generative report drafting or conversational diagnostic workflows. The prompt corpus and candidate pools are restricted to English-language pathology terminology and reporting styles.